{"type":"doc","content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{},"macroMetadata":{"macroId":{"value":"dcadfe6c-94f9-4223-8cc8-3fed295d330c"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"1c2bbf89-5fa6-4e48-92ef-341c583e1b39"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"What is Fluentd?","type":"text"}]},{"type":"paragraph","content":[{"text":"Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data (verbatim from Fluentd documentation). For more information on Fluentd, refer to its documentation ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.fluentd.org/"}}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Integrating Fluentd with Kloudfuse stack","type":"text"}]},{"type":"paragraph","content":[{"text":"Refer to installation instructions for Fluentd ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.fluentd.org/installation"}}]},{"text":". To configure Fluentd to send data to Kloudfuse stack, you can modify the agent’s config or ","type":"text"},{"text":"values.yaml","type":"text","marks":[{"type":"code"}]},{"text":" (if you’re going to install using ","type":"text"},{"text":"helm","type":"text","marks":[{"type":"code"}]},{"text":").","type":"text"}]},{"type":"paragraph","content":[{"text":"You’ll need to configure HTTP output plugin to forward data to Kloudfuse.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"HTTP plugin integration","type":"text"}]},{"type":"paragraph","content":[{"text":"Add the following configuration to Fluentd agent’s config for HTTP configuration.","type":"text"}]},{"type":"codeBlock","content":[{"text":" # Match everything\n @type http\n\n endpoint http://:80/ingester/v1/fluentd\n open_timeout 2\n\n \n @type json\n \n \n chunk_limit_size 1048576 # 1MB\n flush_interval 10s\n \n","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"The above snippet is an example configuration. You can change the buffer configuration to suit your application needs. However the end-point URI path cannot be changed.","type":"text"}]},{"type":"paragraph","content":[{"text":"Currently we support ","type":"text"},{"text":"json","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"msgpack","type":"text","marks":[{"type":"code"}]},{"text":" as the supported types for fluentd HTTP plugin.","type":"text"}]},{"type":"paragraph","content":[{"text":"If you have configured external ingress with TLS for Kloudfuse, use ","type":"text"},{"text":"https://:443/ingester/v1/fluentd","type":"text","marks":[{"type":"code"}]},{"text":" as the endpoint instead.","type":"text"}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Configuration","type":"text"}]},{"type":"paragraph","content":[{"text":"Kloudfuse UI allows you to filter log events based on log labels/tags. You’ll find the label selectors and filter on the left nav bar of the UI. To get a seamless experience with Kloudfuse while using Fluentd agent, we recommend the following configuration(s) or customization(s).","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Kubernetes Labels","type":"text"}]},{"type":"paragraph","content":[{"text":"Fluentd has a filter called ","type":"text"},{"text":"kubernetes","type":"text","marks":[{"type":"code"}]},{"text":" which will enrich the log event with Kubernetes metadata. Refer to the documentation on this filter ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter"}}]},{"text":". If you have application deployed in a Kubernetes environment, we highly recommend enabling this filter for all those applications. Here’s an example configuration for this filter:","type":"text"}]},{"type":"codeBlock","content":[{"text":" # Match everything\n @type kubernetes_metadata\n @id filter_kube_metadata_catalog\n skip_labels false\n skip_container_metadata false\n skip_namespace_metadata false\n skip_master_url true\n ","type":"text"}]},{"type":"codeBlock"},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"If you’re going to add any of our recommended filters, ensure that they don’t conflict with the existing filter definitions.","type":"text"}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Log source","type":"text"}]},{"type":"paragraph","content":[{"text":"By default, Kloudfuse stack looks for ","type":"text"},{"text":"container_name","type":"text","marks":[{"type":"code"}]},{"text":" in the Fluentd payload as the log source. However, this will only be populated if the Fluentd agent is configured with ","type":"text"},{"text":"kubernetes","type":"text","marks":[{"type":"code"}]},{"text":" filter. If you want to Kloudfuse stack to use a different key as the log source, then include the following section under ","type":"text"},{"text":"logs-parser","type":"text","marks":[{"type":"code"}]},{"text":" section in your custom Kloudfuse’s ","type":"text"},{"text":"values.yaml","type":"text","marks":[{"type":"code"}]}]},{"type":"codeBlock","content":[{"text":"kf_parsing_config:\n config: |-\n - remap:\n args:\n kf_source:\n - \"$.\" # must be JSONPath\n conditions:\n - matcher: \"__kf_agent\"\n value: \"fluentd\"\n op: \"==\"","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Log message","type":"text"}]},{"type":"paragraph","content":[{"text":"Fluentd agent includes the log event message in ","type":"text"},{"text":"log","type":"text","marks":[{"type":"code"}]},{"text":" key in the payload. However this can be overriden in the agent configuration. You can customize key which Kloudfuse stack should look for to get the log event message. To customize this setting, include the following section under ","type":"text"},{"text":"logs-parser","type":"text","marks":[{"type":"code"}]},{"text":" section in your custom Kloudfuse’s ","type":"text"},{"text":"values.yaml","type":"text","marks":[{"type":"code"}]}]},{"type":"codeBlock","content":[{"text":"kf_parsing_config:\n config: |-\n - remap:\n args:\n kf_msg:\n - \"$.\" # must be JSONPath\n conditions:\n - matcher: \"__kf_agent\"\n value: \"fluentd\"\n op: \"==\"","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Kloudfuse stack already looks for log message with these key names in the payload: ","type":"text"},{"text":"\"log\", \"LOG\", \"Log\", \"message\", \"msg\", \"MSG\", \"Message\"","type":"text","marks":[{"type":"code"}]}]}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":4},"content":[{"text":"Agent extracted key-value pairs","type":"text"}]},{"type":"paragraph","content":[{"text":"Fluentd supports various parsers to extract key value pairs from an unstructured log. For a full list of parsers, refer to the documentation ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.fluentd.org/parser"}}]},{"text":". By default, Kloudfuse will add all these key-value pairs to ","type":"text"},{"text":"log facets","type":"text","marks":[{"type":"link","attrs":{"href":"https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/752713744/Logs#Log-Facets-filter"}}]},{"text":", which can be filtered on the UI. Note that Kloudfuse cannot differentiate between these key-value pairs and any metadata fields, added by any filter other than ","type":"text"},{"text":"kubernetes","type":"text","marks":[{"type":"code"}]},{"text":". However, you can customize Kloudfuse stack by adding a list of prefix labels. To customize this setting, include the following section under ","type":"text"},{"text":"logs-parser","type":"text","marks":[{"type":"code"}]},{"text":" section in your custom Kloudfuse’s ","type":"text"},{"text":"values.yaml","type":"text","marks":[{"type":"code"}]}]},{"type":"codeBlock","content":[{"text":"kf_parsing_config:\n config: |-\n - remap:\n args:\n kf_additional_tags:\n - \"$.\" # must be JSONPath\n conditions:\n - matcher: \"__kf_agent\"\n value: \"fluentd\"\n op: \"==\"","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"kf_additional_tags","type":"text","marks":[{"type":"code"}]},{"text":" is a list of key prefixes. So, any key that is a prefix match at the top level json will be included as a log label/tag, and not included as a log facet.","type":"text"}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"You don’t need to include any keys specified in ","type":"text"},{"text":"log_source","type":"text","marks":[{"type":"code"}]},{"text":" or ","type":"text"},{"text":"message","type":"text","marks":[{"type":"code"}]},{"text":" or metadata fields from ","type":"text"},{"text":"kubernetes","type":"text","marks":[{"type":"code"}]},{"text":" filter. They’re automatically treated as metadata and not included as a log facet.","type":"text"}]}]}],"version":1}

Browser not supported