Integrating Kloudfuse ingester with fluent-bit

1 Architecture
2 Configuration
- 2.1 Enabling fluent-bit deployment
- 2.2 Routing rules
- 2.3 Add/Remove/Modify fields
- 2.4 Monitoring
3 References

Kloudfuse provides an option to integrate Kloudfuse ingester with fluent-bit to ingest logs from Kinesis Firehose datastream. This option enables you to forward logs to a different datastore/backend other than Kloudfuse. For instance, if you want to forward logs to Elastic search, you can use this option to forward logs to it with appropriate fluent-bit configuration.

Architecture

Figure 1 depicts a rough architectural diagram on how Kloudfuse provides this integration.

Figure 1

At a high level, you’ll have to configure and define log streams in AWS Kinesis Firehose data stream, and connect it to a AWS Cloudwatch subscription. Refer to this doc^[1] on how to configure/stream AWS Kinesis firehose for Cloudwatch logs to Kloudfuse stack. The incoming data will be forwarded from a nginx load balancer (deployed as part of Kloudfuse stack) to ingester service. Ingester is deployed as a Kubernetes deployment within Kloudfuse stack. The ingester will unpack the data and stream individual log events to a fluent-bit service, also deployed as part of the Kloudfuse stack. Fluent-bit is also deployed as deployment within the Kloudfuse stack, and deployed with a HTTP plugin^[2] to accept incoming traffic locally within the Kubernetes cluster. Fluent-bit will then forward the incoming log events to Elasticsearch cluster, using ES plugin^[3].

Configuration

Enabling fluent-bit deployment

Kloudfuse stack independently supports integrating with fluent-bit agent^[4]. The configuration described below is specific to supporting Kloudfuse integration for Elasticsearch using fluent-bit.

Follow the steps below to enable fluent-bit deployment and configuring ingester to forward traffic to fluent-bit:

Add the following section to global values in your custom-values.yaml file^[5].

  kfuse-fbit:
    enabled: true

Add this to ingester configuration in your custom-values.yaml

ingester:
  config:
    logs:
      msgFormat: fluent-bit
    # Possible value for 'msgFormat' are 'kfuse', 'fluent-bit' or 'all'
    # The default value is 'kfuse'. 'all' will forward logs to both Kloudfuse
    # backend and Elasticsearch backend.

Enable fluent-bit deployment using this configuration

fluent-bit:
  # Add any affinity rules, if you have affinity setup in your Kubernetes cluster.
  affinity: {}
  # Add any tolerations, if you have them enabled in your Kubernetes cluster.
  tolerations: []
  config:
    inputs: |-
      [INPUT]
          Name   http
          listen 0.0.0.0
          # default port is 9880. If you want to override it, uncomment line
          # below, update the value
          # port 9880
    outputs: |-
      [OUTPUT]
          Name es
          Match *
          Host <ES_HOST>
          Index <ES_INDEX_NAME>
          HTTP_User <ES_USER>
          HTTP_Passwd <ES_PASSWD>
          Retry_Limit False
          compress gzip
          Suppress_Type_Name On
  extraPorts:
  # ensure that this port matches the port defined in the input HTTP plugin.
  # If you're going with the default value, you don't need to change anything
  # below.
  - containerPort: 9880
    name: in-http
    port: 9880
    protocol: TCP
    targetPort: 9880
  imagePullSecrets:
  - name: kfuse-image-pull-credentials
  kind: Deployment

There are placeholder values in the configuration above, surrounded by <>. Ensure that you replace them with appropriate values, before using them.

Routing rules

Fluent-bit agent routes log events based on tags^[6]. If you want to forward logs to different Elasticsearch hosts/indices, you can use rewrite_tag^[7] filter to do that. You can add as many rewrite_tag plugins as required for your use-case.

If the rule defined inrewrite_tag filter is applied, fluent-bit will re-ingest the log event with the new tag from the beginning of the pipeline. So ensure that you don’t accidentally write multiple rules, that will end in an infinite loop. fluent-bit will not complain about this configuration, but will cause runtime issues.

Add/Remove/Modify fields

Fluent-bit also provides a way to add/remove fields using record_modifier^[8] filter. You can add/define as many filters as you want,

similar to rewrite_tag filter.

Monitoring

Fluent-bit can also report prom-style metrics. To enable collection/scraping of these metrics and ingest them into Kloudfuse stack, add the following configuration to your custom_values.yaml.

This configuration has some common keys that are shared with the basic configuration. Make sure you merge them properly.

Fluent-bit publishes metrics across various input, filter, output plugins including number of records processed and dropped per plugin^[9].

References

[1] - Collection of AWS CloudWatch Logs and Metrics | CloudWatch Logs

[2] - HTTP | Fluent Bit: Official Manual

[3] - Elasticsearch | Fluent Bit: Official Manual

[4] - Fluent Bit Integration

[5] - Installation | Installation

[6] - https://docs.fluentbit.io/manual/concepts/key-concepts#tag

[7] - https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag

[8] - https://docs.fluentbit.io/manual/pipeline/filters/record-modifier

[9] - https://docs.fluentbit.io/manual/administration/monitoring#metric-descriptions