Assumptions

• If using an existing dd-agent version (and just pointing it to KloudFuse instead of DD HQ)

  • dd agent version needs to be 7.41 or higher.

  • validate using following steps:

    • check the chart version with helm list - should be 3.1.10 or higher

      Code Block
      helm list -n <namespace-where-agent-is-installed>

    • check the image version of the agent with describe pod on dd-agent pod.

      Code Block
      k describe pod -n <namespace-where-agent-is-installed> | grep Image Image: gcr.io/datadoghq/agent:7.36.0

Install DataDog Agent

Required Datadog configurations

...

1. Update dd-values-kfuse.yaml file to include following for the custom tags (please note that each tag should be added as key:value as a string separated by the ":") :

datadog:
  tags:
    - custom_tag_name: custom_tag_value

2. kfuse helmcustom-values.yaml file needs to be updated to whitelist the new custom tag (note that the following entry needs to be “appended” to the existing list, without which some default values will get overwritten which is not the desired behavior. Please contact the Kloudfuse team for assistance):

ingester:
  config:
    hostTagIncludes:
    - kf
    - kfuse
    - kube_cluster_name
    - kubernetes.io/hostname
    - node.kubernetes.io/instance-type
    - org_id
    - project
    - topology.kubernetes.io/region
    - topology.kubernetes.io/zone
    ...
    - custom_tag_name

...

Enabling Pods to be detected by Prometheus Autodiscovery

In addition to prometheusScrape to be enabled in the datadog values yaml, the pods needs to have the following annotations. Note that if the application pods are deployed using helm, typically the helm values support a podAnnotations section.

  prometheus.io/path: <specify prometheus endpoint path, e.g., /metrics>
  prometheus.io/port: <SPECIFY promethus endpoint here, e.g., "9090">
  prometheus.io/scrape: "true"

Metadata for metrics collected using the openmetrics check

If using the above configuration, then the openmetrics check is enabled in the agent. See here for more details on what openmetrics check does. Kloudfuse agent (installed using the provided values file) employs a custom check (kf_openmetrics) to collect metadata (the “Description” and “Type” of metrics) for metrics collected using the openmetrics check, which. by default, doesn’t collect any metadata. To enable collection of metadata for these metrics, the sources are required to be annotated which will enable the agent auto-discovery for these pods and execute the custom check.

Kubernetes environment

To enable metrics metadata to be collected from a Kubernetes pod (as shown in an example here) follow these steps (Note that this can be done in helm as well for each of the deployment/sts):

apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<POD_NAME>'
annotations:
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check_names: '["kf_openmetrics"]'
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.init_configs: '[{}]'
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances: '[ { "openmetrics_endpoint": "<http://%%host%%:%%port%%/metrics>"}
      ]'
    # (...)
spec:
  containers:
    - name: '<CONTAINER_IDENTIFIER>'
# (...)

Advance monitoring using Knight

Enable kubernetes_state_metrics

Advance monitoring (Kfuse 1.3 or higher) currently has dependency on kubernetes_state_metrics (KSM) check which is not enabled in the newer version of the agent (2.0) by default. Please ensure that the agent continues to capture these metrics through KSM. To do that, please add/update the dd-agent values file as follows:

datadog:
  kubeStateMetricsEnabled: true
  kubeStateMetricsCore:
    enabled: true
    ignoreLegacyKSMCheck: false

Enable Knight based monitoring in kfuse

Add knightEnabled in the custom-values.yaml and then upgrade the cluster.

ingester:
  ...
  config:
    knightEnabled: true