Traces Integrations for Datadog, OpenTelemetry, and Elastic APM Agent

Kloudfuse supports OpenTelemetry standard for Application Performance Management (APM) and distributed tracing. However, Kloudfuse allows you to keep your existing instrumentation and agents/collectors for distributed tracing. The following table lists the various supported instrumentation/SDKs and agents/collectors.

Instrumentation/SDK	Agent/Collector	Configuration

Instrumentation/SDK	Agent/Collector	Configuration
Datadog	Datadog Agent	Enable apm and point `apm_dd_url` to kloudfuse ingester endpoint
OpenTelemetry	OpenTelemetry collector	otlphttp exporter
Elastic APM Agent (all languages)	No additional collector required	Point ELASTIC_APM_SERVER_URL environment variable to kloudfuse ingester endpoint

OpenTelemetry (Otel) Collector:

Helm Installation:

Get the IP for the Kloudfuse endpoint with the following kubectl command (note: if kfuse stack is installed with a DNS or VPC Endpoint then use that address instead of IP address):

kubectl get svc -n kfuse | grep kfuse-ingress-nginx-controller-internal
kfuse-ingress-nginx-controller-internal   LoadBalancer   10.53.250.80    10.53.232.3   80:32716/TCP,443:30767/TCP   125m

Create a file called otel-values.yaml containing following (note the use of the IP address from above and therefore the tls.insecure set to true. By default this is always true) and replace the endpoint accordingly. Ensure that https or http is used depending if Kfuse has tls enabled or disabled.

image:
  repository: "otel/opentelemetry-collector-contrib"

command:
  name: "otelcol-contrib"
  
config:
  exporters:
    logging:
      verbosity: basic
    otlphttp:
      tls:
        insecure: true # add only if you're using insecure communication 
      metrics_endpoint: https://<REPLACE KFUSE ADDRESS>/ingester/otlp/metrics
      traces_endpoint: https://<REPLACE KFUSE ADDRESS>/ingester/otlp/traces
  extensions:
    health_check: {}
    memory_ballast:
      size_in_percentage: 40
  processors:
    batch:
      timeout: 10s
    memory_limiter:
      check_interval: 5s
      limit_percentage: 80
      spike_limit_percentage: 25
    resourcedetection:
      detectors:
        - env
        - eks
        - ec2
        - gcp
        - aks
        - azure
      override: false
      timeout: 2s
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          cors:
            allowed_origins:
              - http://*
              - https://*
          endpoint: 0.0.0.0:4318
  service:
    extensions:
      - health_check
      - memory_ballast
    pipelines:
      metrics:
        exporters:
          - logging
          - otlphttp
        processors:
          - memory_limiter
          - batch
          - resourcedetection
        receivers:
          - otlp
      traces:
        exporters:
          - otlphttp
        processors:
          - batch
          - resourcedetection
        receivers:
          - otlp
    telemetry:
      metrics:
        address: ${MY_POD_IP}:8888
mode: daemonset
nameOverride: otelcol
ports:
  metrics:
    enabled: true
  otlp:
    enabled: true
  otlp-http:
    enabled: true
presets:
  kubernetesAttributes:
    enabled: true

Please add any tolerations & affinity as required with your deployment.

Install collector with following steps using the created otel-values.yaml file

# Add otel repo and update
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
# Install otel collector:
helm install my-opentelemetry-collector open-telemetry/opentelemetry-collector -f otel-values.yaml

Note: The above instructions install the collector in daemonset mode. Please review the otel-collector documentation for additional modes and more configuration options.

Datadog Agent:

Configure the datadog agent with apm_dd_url set to the Kloudfuse ingester. You can find the Kloudfuse Platform ingress internal IP with the following kubectl command (note: if kfuse stack is installed with a DNS or VPC Endpoint then use that address instead of IP address):

In the Datadog agent yaml, add the apm_config section with apm_dd_url pointing to the Kloudfuse ingester.

Datadog Agent Runtime Metrics

To collect runtime metrics from the application and emit them using the DD agent, make use of the DogStatsD service which is used for custom metrics collection and emitting.

For proper function of the service ensure the following settings:

To correctly send runtime metrics with the appropriate k8s labels such as namespace, add the following environment variable to your application service:

External

Traces Integrations for Datadog, OpenTelemetry, and Elastic APM Agent

Related pages