Configure Ingestion Authentication
- Abhishek Chaturvedi (Unlicensed)
- Suryakant Kashyap
- Roza Leyderman
To enable authentication for ingestion, follow these 3 steps. For step 2 choose the section relevant to the agents being used.
Step 1. Generate AUTH_TOKEN
Generate an auth token (referred to as AUTH_TOKEN) and store this value in a safe location. You will need to use this later in more than one place.
AUTH_TOKEN=`cat /dev/urandom | env LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1`Base64 encode the AUTH_TOKEN (referred to as AUTH_TOKEN_ENCODED )
AUTH_TOKEN_ENCODED=`echo -n $AUTH_TOKEN | base64`Step 2. Configure Telemetry agents/sources
Follow instructions for the corresponding sources below.
AWS CloudWatch metrics & Logs (Kinesis)
Prometheus Remote Write
Update prometheus remote write configuration as shown below:
prometheus.yml:
remote_write:
- url: https://<customer>.kloudfuse.io/ingester/write
authorization:
credentials: <AUTH_TOKEN>If you’re using prometheus operator, please refer to the configuration below
Fluent Bit
Update/Add the following Headers field with AUTH_TOKEN replaced with the one generated in step 1, in the HTTP plugin section of the fluent-bit configuration file as shown below:
Using Kubernetes Secret for setting the Kf-Api-Key
Create a secret as following
Update the fluent-bit helm custom-values.yaml as below
Fluentd
Update/Add the fluentd output http plugin configuration to add a “headers" field as described below using the Kf-Api-Key and AUTH_TOKEN:
Filebeat
Update/Add the filebeat configuration to include the api_key field within the output section:
OLTP Collector for Metrics/Logs/Traces
Update/Add following headers section in the exporters section.
DD/Kfuse agent
Update/Add the dd-agent configuration file to add the AUTH_TOKEN as the apiKey as shown below:
Azure Event Hub
Update the index.js file in your FunctionApp's Trigger Function to add the AUTH_TOKEN as the KF_API_KEY as shown below:
To configure sending logs to your kloudfuse platform, follow the steps on this wiki page.
AWS CloudWatch metrics & Logs (Kinesis)
When configuring kinesis firehose data stream to send logs/metrics from Cloudwatch, use the AUTH_TOKEN value generated in step 1 as the access token. If the firehose data stream is already setup, then update it to use AUTH_TOKEN value as access token.
AWS Eventbridge Events
When configuring Eventbridge to ingest to Kloudfuse, use AUTH_TOKEN as the value for the Kf_Api_Key.
Step 3: Configure kfuse
Use the base64 encoded value of the AUTH_TOKEN (AUTH_TOKEN_ENCODED) and create a kubernetes secret with the name kfuse-auth-ingest:
Multiple Authorization Keys
New feature available in Release 2.7.2
You can configure multiple authorization tokens inside the secret. These tokens can contain any string value, and can be used as a human-readable identifier to reference the auth token.
Here, line 8 specifies the second key, authkey2.
Remember to update the custom-values.yaml file to enable ingestion authentication:
Configuration of Additional Labels based on Auth Token
From Kloudfuse 2.7.2 onwards, additional labels can be configured to be attached to MELT data based on the auth token used by the incoming payload.
To configure these labels, add the following in the custom-values.yaml (replace accordingly). Take note that the same keys are used in the secret and the ingester config.