Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Next »

Kfuse Knight collects and aggregates dynamic data on nodes, using eBPF. It runs as a daemonset on a Kubernetes cluster. It can automatically trace protocol-specific network traffic to/from the containers to generate RED metrics. By default, http protocol tracing is enabled. Please contact us, for enabling collection of other protocols.

The RED metrics are exposed as a Prometheus metrics endpoint from the Kfuse Knight pod and can be discovered/scraped by Prometheus and/or Datadog agent.

Install Instructions

# Create a namespace.
kubectl create namespace kfuse-knight
# Get the container registry key from kloudfuse <token>.json -- (uploaded to the page)
# Make sure to create a k8s secret to use this registry access token which helm will use to upgrade/install
kubectl config set-context --current --namespace=kfuse-knight

# On ubuntu
cat token.json | docker login -u _json_key  --password-stdin https://us-east1-docker.pkg.dev
kubectl create secret docker-registry kfuse-image-pull-credentials \
        --namespace='kfuse-knight' --docker-server 'us.gcr.io' --docker-username _json_key \
        --docker-email 'container-registry@mvp-demo-301906.iam.gserviceaccount.com' \
        --docker-password=''"$(cat token.json)"''

# On windows
type token.json | helm registry login -u _json_key  --password-stdin https://us-east1-docker.pkg.dev

$service_json = (Get-Content token.json).replace("\n", "").replace('"', '\"')
kubectl create secret docker-registry kfuse-image-pull-credentials --namespace=kfuse-knight --docker-server 'us.gcr.io' --docker-username _json_key --docker-email 'container-registry@mvp-demo-301906.iam.gserviceaccount.com' --docker-password=""$service_json""
        
# Get the latest knight chart version
helm upgrade --install knight oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse-knight --version=0.1.0-acb11f5 -n kfuse-knight

Customizing the metrics to emit

The Kfuse-Knight helm values can be customized to control protocols traced by knight and any information extracted and tagged into the generated metrics. The config section of the helm values contains a section of what gets extracted by default. It can be adjusted accordingly.

config:
  http:
    metrics:
      # The tags to include/exclude when emitting logs
      # Exact field names can be specified.
      # Regex pattern can be specified.
      # If both includes and excludes are given, then a field is only included
      # if it matches an includes rule and does not match an exclude rule..
      # If a field is json formatted, then subfields can be extracted using
      # the following format: field_name$JSON$<json pointer>
      tags_include:
        - request_type
        - response_code
        - request_path
        - request_headers$JSON$/Accept
        - request_headers$JSON$/Host
      tags_exclude:

Scraping of Metrics

Since Knight exposes a prometheus endpoint, autodiscovery of prometheus endpoint using Datadog or Prometheus should work.

If using Datadog, then the configurations specified in Collecting Data Using Datadog Agent already enables collection of Knight metrics. At the minimum, the following config is needed:

datadog:
  prometheusScrape:
    enabled: true
    version: 1
    additionalConfigs:
      - configurations:
        - send_monotonic_counter: false
          send_distribution_counts_as_monotonic: false
          send_distribution_sums_as_monotonic: false
          send_histograms_buckets: true
          max_returned_metrics: 999999
          min_collection_interval: 15
        autodiscovery:
          kubernetes_annotations:
            exclude:
              app: knight
              prometheus.io/scrape: "false"
      - configurations:
        - send_monotonic_counter: false
          send_distribution_counts_as_monotonic: false
          send_distribution_sums_as_monotonic: false
          send_histograms_buckets: true
          max_returned_metrics: 999999
          min_collection_interval: 60
        autodiscovery:
          kubernetes_container_names:
            - knight

Enable kubernetes_state_metrics

knight based functionality in the kfuse stack (see here) currently requires certain metrics which are emitted by the kubernetes_state_metrics (KSM) check which isn’t installed in the newer version of the agent (2.0) by default. Please ensure that the agent continues to capture these metrics through KSM. To do that, please add/update the dd-agent values file as follows: 

datadog:
  kubeStateMetricsEnabled: true
  kubeStateMetricsCore:
    enabled: true
    ignoreLegacyKSMCheck: false
  • No labels