/
Metrics Ingestion using Pushgateway

Metrics Ingestion using Pushgateway

Owned by Ayan Shah
Last updated: Dec 27, 2024
2 min read

Kloudfuse 3.1.0 or higher supports ingesting metrics using Prometheus-Pushgateway

The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus. They can instead push their metrics to a Pushgateway and it then exposes these metrics to Prometheus.

Enable Pushgateway on Kloudfuse

You will need to update your values.yaml file used for Kfuse helm chart installation to enable the pushgateway:

global: orgId: "..." # ... pushgateway: enabled: true # Needed to scrape and send metrics to Kfuse cloud-exporter: enabled: true # ...

After updating the values.yaml, you can install/upgrade your Kfuse release using the following command:

helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse --version 3.1.0 -f values.yaml

Once the upgrade is done, you will see the pod for kfuse-pushgateway spin up in your env.

 

Sending metrics to the Pushgateway

Using the Prometheus text protocol, pushing metrics is easy so no separate CLI is provided. Simply use a command-line HTTP tool like curl. Your favorite scripting language has most likely some built-in HTTP capabilities you can leverage here as well.

Note that in the text protocol, each line has to end with a line-feed character (aka 'LF' or '\n'). Ending a line in other ways, e.g. with 'CR' aka '\r', 'CRLF' aka '\r\n', or just the end of the packet, will result in a protocol error.

Pushed metrics are managed in groups, identified by a grouping key of any number of labels, of which the first must be the job label. The groups are easy to inspect via the web interface.

For implications of special characters in label values see the URL section below.

Examples:

  • Push a single sample into the group identified by {job="some_job"}:

    echo "some_metric 3.14" | curl --data-binary @- https://<KFUSE-DNS>/metrics/job/some_job

    Since no type information has been provided, some_metric will be of type untyped.

  • Push something more complex into the group identified by {job="some_job",instance="some_instance"}:

    Note how type information and help strings are provided. Those lines are optional, but strongly encouraged for anything more complex.

For more information refer here

 

Once the metrics are flowing into pushgateway, our kfuse-scraper scrapes it to ingest metrics into the Kloudfuse database and they should appear on the Kfuse Metrics summary.