Versions Compared

Version Old Version 3 New Version 4
Changes made by

Vinay Krishnamurthy

Vinay Krishnamurthy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

With DD plugin

...

What is Fluent Bit?

Fluent Bit is an open source and multi-platform log processor tool which aims to be a generic Swiss knife for logs processing and distribution (verbatim from Fluent Bit’s documentation). For more information on Fluent Bit, refer to the documentation here.

Integrating Fluent Bit with Kloudfuse stack

Fluent Bit supports a variety of input and output plugins and parsers. For more information on the supported output plugins, see here. However, Kloudfuse stack currently supports 2 output plugins:

  • DD (DataDog) plugin

  • HTTP plugin

You can install Fluent Bit using helm charts. Clone the charts repo from here by running:

Code Block
git clone https://github.com/fluent/helm-charts.git

You’ll now need to configure the Fluent Bit installation by modifying charts/fluent-bit/values.yaml depending on the plugin you choose to use.

DD plugin integration

Add the following content values.yaml, if you’re planning on using DD output plugin with fluent-bitFluent Bit:

Code Block
  outputs: |
    [OUTPUT]
        Name datadog
        Match <match_pattern>
        Host <kfuse_ingress_ip>
        dd_message_key log
        TLS on
        compress gzip
        apiKey random
        dd_tags <additional_tags>
        dd_source <dd_source_name>
        dd_service <dd_service_name>

...

Info

All the settings/configuration up to TLS on are required for the kfuse stack to work properly. The rest of the settings are recommended.

...

HTTP plugin integration

If you’re not planning on using DD output plugin, then you can use the HTTP plugin to push the logs to kfuse stack. Use Add the following configuration to values.yaml for HTTP plugin:configuration.

Code Block
    [OUTPUT]
        Name http
        Match <match_pattern>
        Host <kfuse_ingress_ip>
        Port 80
        URI /ingester/v1/fluent_bit
        Format msgpack

...

Info

Currently msgpack is the only supported format for HTTP plugin. You can also not include this setting. The default format for HTTP plugin is msgpack.

Installing Fluent Bit helm chart

Info

You will only need to edit the values.yaml file to configure either DD or HTTP plugin. You do not have to configure both.

After modifying the values.yaml file, run the following two steps to install Fluent Bit:

Code Block
helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit -f charts/fluent-bit/values.yaml

Wait for the Fluent Bit pods to come up, then verify that the logs are being pushed Kloudfuse stack by logging into the Kloudfuse UI.