Collection of AWS ECS/ECS Fargate Container Logs with Fluent Bit

An Amazon Elastic Container Service (Amazon ECS) task definition allows you to specify only a single log configuration object for a given container. This limit means that you can forward logs to only a single destination. To forward logs to multiple destinations in Amazon ECS on Fargate, you can use FireLens.

Create a Custom Fluent Bit docker image

  1. Create a custom Fluent Bit configuration file called logDestinations.conf with your choice of [OUTPUT] definitions. For example, the following configuration file includes configuration defined for KloudFuse (see Output for more examples):

[OUTPUT] Name http Match * Host <Kloudfuse DNS> Port 443 TLS on TLS.debug 4 URI /ingester/v1/fluent_bit Format {json|json_lines|json_stream|msgpack}
  1. Use the following Dockerfile example to create a Docker image with a custom Fluent Bit output configuration file:

    FROM amazon/aws-for-fluent-bit:latest ADD logDestinations.conf /logDestinations.conf
  2. To use the Dockerfile that you created to create the custom fluent-bit Docker image, run the following command:

    docker build -t custom-fluent-bit:latest .
  3. Push your image to the ECR Image repo where the ECS Containers will be pulling the images from

 

Deploy Fluent Bit Container in ECS

  1. In your task definition, update the options for your FireLens configuration:

 

For more details ECS FluentBit task definition with an example see AWS Doc here