Consumption Tracking for MELTA

Kfuse stack can be configured to track per stream consumption. This allows the consumer to track the usage of the stack based on ingested labels. To do so, please follow these instructions.

update custom-values.yaml to include the “trackingLabels" which should be used by kfuse. As an example, if you the consumption needs to be tracked using “kube_cluster_name" & "kube_namespace", specify the following in the config file.

ingester:
  ...
  config:
    trackingLabels:
    - kube_cluster_name
    - kube_namespace

logs-parser:
  ...
  extraApplicationConfigs:
      trackingLabels = [kube_cluster_name, kube_namespace]

Note that this creates additional labels (with a target_ prefix, which can be configured as well, added to the tracking labels) for each incoming sample of individual stream. With the above configuration, kfuse can track following metrics for each of the MELT streams:

Amount of data ingested in bytes for each of the MELT stream over a duration per tracking labels combination.
Storage consumed over any duration for each of the MELT stream per tracking labels combination.
# of queries as well as amount of data scanned for each of the MELT stream per tracking label combinations.

A default dashboard for consumption tracking (Consumption Tracking) is available in kfuse provided dashboards.