Traces Integrations

Kloudfuse supports OpenTelemetry standard for Application Performance Management (APM) and distributed tracing. However, Kloudfuse allows you to keep your existing instrumentation and agents/collectors for distributed tracing. The following table lists the various supported instrumentation/SDKs and agents/collectors.

Instrumentation/SDK

Agent/Collector

Configuration

Instrumentation/SDK

Agent/Collector

Configuration

Datadog

Datadog Agent

Enable apm and point apm_dd_url to kloudfuse ingester endpoint

OpenTelemetry

OpenTelemetry collector

otlphttp exporter

Elastic APM Agent (all languages)

No additional collector required

Point ELASTIC_APM_SERVER_URL environment variable to kloudfuse ingester endpoint

OpenTelemetry (Otel) Collector:

Helm Installation:

  • Get the IP for the Kloudfuse endpoint with the following kubectl command (note: if kfuse stack is installed with a DNS or VPC Endpoint then use that address instead of IP address):

kubectl get svc -n kfuse | grep kfuse-ingress-nginx-controller-internal kfuse-ingress-nginx-controller-internal LoadBalancer 10.53.250.80 10.53.232.3 80:32716/TCP,443:30767/TCP 125m
  • Create a file called otel-values.yaml containing following (note the use of the IP address from above and therefore the tls.insecure set to true. By default this is always true) and replace the endpoint accordingly. Ensure that https or http is used depending if Kfuse has tls enabled or disabled.

image: repository: "otel/opentelemetry-collector-contrib" command: name: "otelcol-contrib" config: exporters: logging: verbosity: basic otlphttp: tls: insecure: true # add only if you're using insecure communication metrics_endpoint: https://<REPLACE KFUSE ADDRESS>/ingester/otlp/metrics traces_endpoint: https://<REPLACE KFUSE ADDRESS>/ingester/otlp/traces extensions: health_check: {} memory_ballast: size_in_percentage: 40 processors: batch: timeout: 10s memory_limiter: check_interval: 5s limit_percentage: 80 spike_limit_percentage: 25 resourcedetection: detectors: - env - eks - ec2 - gcp - aks - azure override: false timeout: 2s receivers: otlp: protocols: grpc: endpoint: ${env:MY_POD_IP}:4317 http: cors: allowed_origins: - http://* - https://* endpoint: 0.0.0.0:4318 service: extensions: - health_check - memory_ballast pipelines: metrics: exporters: - logging - otlphttp processors: - memory_limiter - batch - resourcedetection receivers: - otlp traces: exporters: - otlphttp processors: - batch - resourcedetection receivers: - otlp telemetry: metrics: address: ${MY_POD_IP}:8888 mode: daemonset nameOverride: otelcol ports: metrics: enabled: true otlp: enabled: true otlp-http: enabled: true presets: kubernetesAttributes: enabled: true

Please add any tolerations & affinity as required with your deployment.

  • Install collector with following steps using the created otel-values.yaml file

# Add otel repo and update helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm repo update # Install otel collector: helm install my-opentelemetry-collector open-telemetry/opentelemetry-collector -f otel-values.yaml

Note: The above instructions install the collector in daemonset mode. Please review the otel-collector documentation for additional modes and more configuration options.

 

Datadog Agent:

Configure the datadog agent with apm_dd_url set to the Kloudfuse ingester. You can find the Kloudfuse Platform ingress internal IP with the following kubectl command (note: if kfuse stack is installed with a DNS or VPC Endpoint then use that address instead of IP address):

In the Datadog agent yaml, add the apm_config section with apm_dd_url pointing to the Kloudfuse ingester.

Datadog Agent Runtime Metrics

To collect runtime metrics from the application and emit them using the DD agent, make use of the DogStatsD service which is used for custom metrics collection and emitting.

For proper function of the service ensure the following settings:

To correctly send runtime metrics with the appropriate k8s labels such as namespace, add the following environment variable to your application service: