Integrating Kloudfuse ingester with fluent-bit

1 Architecture
2 Configuration
- 2.1 Enabling fluent-bit deployment
- 2.2 Routing rules
- 2.3 Add/Remove/Modify fields
- 2.4 Monitoring
3 References

Kloudfuse provides an option to integrate Kloudfuse ingester with fluent-bit to ingest logs from Kinesis Firehose datastream. This option enables you to forward logs to a different datastore/backend other than Kloudfuse. For instance, if you want to forward logs to Elastic search, you can use this option to forward logs to it with appropriate fluent-bit configuration.

Architecture

Figure 1 depicts a rough architectural diagram on how Kloudfuse provides this integration.

Figure 1

At a high level, you’ll have to configure and define log streams in AWS Kinesis Firehose data stream, and connect it to a AWS Cloudwatch subscription. Refer to this doc^[1] on how to configure/stream AWS Kinesis firehose for Cloudwatch logs to Kloudfuse stack. The incoming data will be forwarded from a nginx load balancer (deployed as part of Kloudfuse stack) to ingester service. Ingester is deployed as a Kubernetes deployment within Kloudfuse stack. The ingester will unpack the data and stream individual log events to a fluent-bit service, also deployed as part of the Kloudfuse stack. Fluent-bit is also deployed as deployment within the Kloudfuse stack, and deployed with a HTTP plugin^[2] to accept incoming traffic locally within the Kubernetes cluster. Fluent-bit will then forward the incoming log events to Elasticsearch cluster, using ES plugin^[3].

Configuration

Enabling fluent-bit deployment

Kloudfuse stack independently supports integrating with fluent-bit agent^[4]. The configuration described below is specific to supporting Kloudfuse integration for Elasticsearch using fluent-bit.

Follow the steps below to enable fluent-bit deployment and configuring ingester to forward traffic to fluent-bit:

Add the following section to global values in your custom-values.yaml file^[5].

  kfuse-fbit:
    enabled: true

Add this to ingester configuration in your custom-values.yaml

ingester:
  config:
    logs:
      msgFormat: fluent-bit
    # Possible value for 'msgFormat' are 'kfuse', 'fluent-bit' or 'all'
    # The default value is 'kfuse'. 'all' will forward logs to both Kloudfuse
    # backend and Elasticsearch backend.

Enable fluent-bit deployment using this configuration

fluent-bit:
  # Add any affinity rules, if you have affinity setup in your Kubernetes cluster.
  affinity: {}
  # Add any tolerations, if you have them enabled in your Kubernetes cluster.
  tolerations: []
  config:
    inputs: |-
      [INPUT]
          Name   http
          listen 0.0.0.0
          # default port is 9880. If you want to override it, uncomment line
          # below, update the value
          # port 9880
    outputs: |-
      [OUTPUT]
          Name es
          Match *
          Host <ES_HOST>
          Index <ES_INDEX_NAME>
          HTTP_User <ES_USER>
          HTTP_Passwd <ES_PASSWD>
          Retry_Limit False
          compress gzip
          Suppress_Type_Name On
  extraPorts:
  # ensure that this port matches the port defined in the input HTTP plugin.
  # If you're going with the default value, you don't need to change anything
  # below.
  - containerPort: 9880
    name: in-http
    port: 9880
    protocol: TCP
    targetPort: 9880