OpenTelemetry Collector on a Docker Environment

When deploying OpenTelemetry Collector on a Docker Environment, refer to the following sample yaml. Note that kf_platform with docker as value is a required resource attribute. Also refer to https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/1100251273 for additional details regarding runtime metrics and https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/984285374 for details on enabling container resource attributes.

receivers:
  otlp:
    protocols:
      grpc:
      http:
        cors:
          allowed_origins:
            - "http://*"
            - "https://*"

exporters:
  otlphttp:
    logs_endpoint: https://<REPLACE WITH KFUSE ADDRESS>/ingester/otlp/v1/logs
    metrics_endpoint: https://<REPLACE WITH KFUSE ADDRESS>/ingester/otlp/metrics
    traces_endpoint: https://<REPLACE WITH KFUSE ADDRESS>/ingester/otlp/traces

processors:
  batch:
    timeout: 10s
  resource:
    attributes:
    - key: kf_platform
      value: "docker"
      action: upsert
  resourcedetection:
    detectors:
    - env
    - docker
    - ec2
    - gcp
    - azure
    override: true
    timeout: 2s

connectors:
  spanmetrics:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource, resourcedetection]
      exporters: [otlphttp]
    metrics:
      exporters: [otlphttp]
      processors: [batch, resource, resourcedetection]
      receivers: [otlp]
    logs:
      exporters: [otlphttp]
      processors: [batch, resource, resourcedetection]
      receivers: [otlp]

Notes on docker resource detector

The docker resource detector ensures that the host machines hostname is used as the host.name resource attributes. This requires the open telemetry collector to have permissions to read from the docker.sock file. When running the open telemetry collector as part of docker-compose, add the following configuration in the section for open telemetry collector.

    volumes:
      ...
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        read_only: True
    user: 10001:<REPLACE GROUP>

The group is the group of the docker.sock file. This can be obtained by running the following command

stat -c '%g' /var/run/docker.sock

Docker and System Level Metrics

The Kfuse APM Services page integrates with docker and system metrics to show related infrastructure metrics.

The metrics can be exported into Kfuse using Datadog or from the OpenTelemetry collector itself.

If using Datadog to export these metrics, refer to https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/857931777

If using OpenTelemetry Collector, the OpenTelemetry Collector needs to be updated with additional configurations. Specifically, enable dockerstats and hostmetrics receivers, add additional kf_metrics_agent resource attributes.

receivers:
 ...
  docker_stats:

  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true

processors:
  batch:
    timeout: 10s
  resource:
    attributes:
    - key: kf_platform
      value: "docker"
      action: upsert
    - key: kf_metrics_agent
      value: "otlp"
      action: upsert
  resourcedetection:
    detectors:
    - env
    - docker
    - ec2
    - gcp
    - azure
    override: true
    timeout: 2s

service:
  pipelines:
   ...
    metrics:
      exporters: [otlphttp]
      processors: [batch, resource, resourcedetection]
      receivers: [otlp, docker_stats, hostmetrics]