Configure sending Azure Logs and Metrics to Kloudfuse

This page describes the setup process to forward your Azure logs and metrics to Kloudfuse:

Forwarding logs to Kloudfuse:

At high level, we have to setup three things on Azure:

Create an Azure Event Hub.
Configure the Kloudfuse-Azure function with an Event Hub trigger to send logs to Kloudfuse.
Set up diagnostic settings to route Azure Activity logs, resource logs, or both to the Event Hub.

Note: Azure Resources can only send data to Event Hubs in the same Azure region.

Setup steps for Azure Logs:

Follow the steps given in the link to create the listed entities below:

Create an Azure Event Hub https://docs.datadoghq.com/logs/guide/azure-logging-guide/?tab=manualinstallation#create-an-azure-event-hub
1. Create an Event Hubs namespace
2. Add an Event Hub to your Event Hubs namespace
3. Configure shared access
Create the Kloudfuse-Azure function https://docs.datadoghq.com/logs/guide/azure-logging-guide/?tab=manualinstallation#create-the-datadog-azure-function
1. Create a function app
2. Configure your function app with the Event Hub connection string
3. Add a new function to your Function App using the Event Hub trigger template
Point your Event Hub trigger to Kloudfuse
1. On the detail page of your Event Hub trigger function, click Code + Test under the Developer side menu.
2. Add the Kloudfuse-Azure function code to the function’s index.js file.
  1. Set KF_URL in the index.js file to your kloudfuse url. For ex - pisco.kloudfuse.io
  2. Set KF_API_KEY in the index.js file by following the next step 3.c
3. Configure auth on ingestion by following the steps here to generate an AUTH_TOKEN and set it as KF_API_KEY.
4. Save the function.
5. Click Integration under the Developer side menu.
6. Click Azure Event Hubs under Trigger and inputs.
7. Confirm the following settings are in place:
  a. Event hub connection is set to the name of your connection string environment variable.
  b. Event parameter name is set to eventHubMessages.
  c. Event hub name is set to the name of your Event Hub.
  d. Event hub cardinality is set to Many.
  e. Event hub data type is left empty.
8. To validate your setup, click Code + Test under the Developer side menu.
9. Click Test/Run and enter a test message in valid JSON format.
10. Find your test message in the Kloudfuse Logs Explorer.
Create Diagnostic Settings for all of your resources in Azure from which you want to send the logs from https://docs.datadoghq.com/logs/guide/azure-logging-guide/?tab=manualinstallation#create-diagnostic-settings

Forwarding Metrics to Kloudfuse:

Kloudfuse automatically pulls metrics from your Azure subscriptions without requiring any setup on the Azure portal.

To configure sending the metrics to Azure, just add the following configuration in your custom-values.yaml file.

Make sure that cloud-exporter is enabled true in the global section.

global:  
  cloud-exporter:
    enabled: true

Create ClientID, ClientSecret, TenantID and subscription. https://docs.datadoghq.com/integrations/guide/azure-manual-setup/?tab=manual#creating-the-app-registration
Add the following section under kfuse-cloud-exporter
1. Replace the redacted values with your actual values from Azure portal.

Note: Metrics can be fetched from multiple subscriptions in Azure. List all of the subscriptions you want to pull metrics from.

kfuse-cloud-exporter:
  azure-metrics-exporter:
    enabled: true
    subscriptions:
      - xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      - xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    secrets:
      AZURE_CLIENT_ID: "xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      AZURE_TENANT_ID: "xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      AZURE_CLIENT_SECRET: "xxxx~xxxxxx"

OR

b. To mount it from a secret instead of writing it plainly in the custom-values.yaml file, then add this:

kfuse-cloud-exporter:
  azure-metrics-exporter:
    enabled: true
    subscriptions:
      - xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    extraEnv:
      - name: AZURE_CLIENT_ID
        valueFrom:
          secretKeyRef:
            key: AZURE_CLIENT_ID
            name: <Secret-name>
      - name: AZURE_TENANT_ID
        valueFrom:
          secretKeyRef:
            key: AZURE_TENANT_ID
            name: <Secret-name>
      - name: AZURE_CLIENT_SECRET
        valueFrom:
          secretKeyRef:
            key: AZURE_CLIENT_SECRET
            name: <Secret-name>

The secret should contain these key-value pairs: