Integrating Kloudfuse ingester with fluent-bit
Kloudfuse provides an option to integrate Kloudfuse ingester with fluent-bit to ingest logs from Kinesis Firehose datastream. This option enables you to forward logs to a different datastore/backend other than Kloudfuse. For instance, if you want to forward logs to Elastic search, you can use this option to forward logs to it with appropriate fluent-bit configuration.
Architecture
Figure 1 depicts a rough architectural diagram on how Kloudfuse provides this integration.
At a high level, you’ll have to configure and define log streams in AWS Kinesis Firehose data stream, and connect it to a AWS Cloudwatch subscription. Refer to this doc[1] on how to configure/stream AWS Kinesis firehose for Cloudwatch logs to Kloudfuse stack. The incoming data will be forwarded from a nginx load balancer (deployed as part of Kloudfuse stack) to ingester service. Ingester is deployed as a Kubernetes deployment within Kloudfuse stack. The ingester will unpack the data and stream individual log events to a fluent-bit service, also deployed as part of the Kloudfuse stack. Fluent-bit is also deployed as deployment within the Kloudfuse stack, and deployed with a HTTP plugin[2] to accept incoming traffic locally within the Kubernetes cluster. Fluent-bit will then forward the incoming log events to Elasticsearch cluster, using ES plugin[3].
Configuration
Enabling fluent-bit deployment
Kloudfuse stack independently supports integrating with fluent-bit agent[4]. The configuration described below is specific to supporting Kloudfuse integration for Elasticsearch using fluent-bit.
Follow the steps below to enable fluent-bit deployment and configuring ingester to forward traffic to fluent-bit:
Add the following section to
global
values in yourcustom-values.yaml
file[5].
kfuse-fbit:
enabled: true
Add this to ingester configuration in your
custom-values.yaml
ingester:
config:
logs:
msgFormat: fluent-bit
# Possible value for 'msgFormat' are 'kfuse', 'fluent-bit' or 'all'
# The default value is 'kfuse'. 'all' will forward logs to both Kloudfuse
# backend and Elasticsearch backend.
Enable fluent-bit deployment using this configuration
fluent-bit:
# Add any affinity rules, if you have affinity setup in your Kubernetes cluster.
affinity: {}
# Add any tolerations, if you have them enabled in your Kubernetes cluster.
tolerations: []
config:
inputs: |-
[INPUT]
Name http
listen 0.0.0.0
# default port is 9880. If you want to override it, uncomment line
# below, update the value
# port 9880
outputs: |-
[OUTPUT]
Name es
Match *
Host <ES_HOST>
Index <ES_INDEX_NAME>
HTTP_User <ES_USER>
HTTP_Passwd <ES_PASSWD>
Retry_Limit False
compress gzip
Suppress_Type_Name On
extraPorts:
# ensure that this port matches the port defined in the input HTTP plugin.
# If you're going with the default value, you don't need to change anything
# below.
- containerPort: 9880
name: in-http
port: 9880
protocol: TCP
targetPort: 9880