/
Integrating Kloudfuse ingester with fluent-bit

Integrating Kloudfuse ingester with fluent-bit

Kloudfuse provides an option to integrate Kloudfuse ingester with fluent-bit to ingest logs from Kinesis Firehose datastream. This option enables you to forward logs to a different datastore/backend other than Kloudfuse. For instance, if you want to forward logs to Elastic search, you can use this option to forward logs to it with appropriate fluent-bit configuration.

Architecture

Figure 1 depicts a rough architectural diagram on how Kloudfuse provides this integration.

Figure 1

At a high level, you’ll have to configure and define log streams in AWS Kinesis Firehose data stream, and connect it to a AWS Cloudwatch subscription. Refer to this doc[1] on how to configure/stream AWS Kinesis firehose for Cloudwatch logs to Kloudfuse stack. The incoming data will be forwarded from a nginx load balancer (deployed as part of Kloudfuse stack) to ingester service. Ingester is deployed as a Kubernetes deployment within Kloudfuse stack. The ingester will unpack the data and stream individual log events to a fluent-bit service, also deployed as part of the Kloudfuse stack. Fluent-bit is also deployed as deployment within the Kloudfuse stack, and deployed with a HTTP plugin[2] to accept incoming traffic locally within the Kubernetes cluster. Fluent-bit will then forward the incoming log events to Elasticsearch cluster, using ES plugin[3].

Configuration

Enabling fluent-bit deployment

Kloudfuse stack independently supports integrating with fluent-bit agent[4]. The configuration described below is specific to supporting Kloudfuse integration for Elasticsearch using fluent-bit.

Follow the steps below to enable fluent-bit deployment and configuring ingester to forward traffic to fluent-bit:

  • Add the following section to global values in your custom-values.yaml file[5].

kfuse-fbit: enabled: true
  • Add this to ingester configuration in your custom-values.yaml

ingester: config: logs: msgFormat: fluent-bit # Possible value for 'msgFormat' are 'kfuse', 'fluent-bit' or 'all' # The default value is 'kfuse'. 'all' will forward logs to both Kloudfuse # backend and Elasticsearch backend.
  • Enable fluent-bit deployment using this configuration

fluent-bit: # Add any affinity rules, if you have affinity setup in your Kubernetes cluster. affinity: {} # Add any tolerations, if you have them enabled in your Kubernetes cluster. tolerations: [] config: inputs: |- [INPUT] Name http listen 0.0.0.0 # default port is 9880. If you want to override it, uncomment line # below, update the value # port 9880 outputs: |- [OUTPUT] Name es Match * Host <ES_HOST> Index <ES_INDEX_NAME> HTTP_User <ES_USER> HTTP_Passwd <ES_PASSWD> Retry_Limit False compress gzip Suppress_Type_Name On extraPorts: # ensure that this port matches the port defined in the input HTTP plugin. # If you're going with the default value, you don't need to change anything # below. - containerPort: 9880 name: in-http port: 9880 protocol: TCP targetPort: 9880