Fluent Bit Collection of AWS ECS/ECS Fargate Container Logs
An Amazon Elastic Container Service (Amazon ECS) task definition allows you to specify only a single log configuration object for a given container. This limit means that you can forward logs to only a single destination. To forward logs to multiple destinations in Amazon ECS on Fargate, you can use FireLens.
Create a Custom Fluent Bit docker image
Create a custom Fluent Bit configuration file called logDestinations.conf with your choice of [OUTPUT] definitions. For example, the following configuration file includes configuration defined for KloudFuse (see Output for more examples):
[OUTPUT]
Name http
Match *
Host <Kloudfuse DNS>
Port 443
TLS on
TLS.debug 4
URI /ingester/v1/fluent_bit
Format {json|json_lines|json_stream|msgpack}Use the following Dockerfile example to create a Docker image with a custom Fluent Bit output configuration file:
FROM amazon/aws-for-fluent-bit:latest ADD logDestinations.conf /logDestinations.confTo use the Dockerfile that you created to create the custom fluent-bit Docker image, run the following command:
docker build -t custom-fluent-bit:latest .Push your image to the ECR Image repo where the ECS Containers will be pulling the images from
docker push aws_account_id.dkr.ecr.region.amazonaws.com/custom-fluent-bit:latest
Deploy Fluent Bit Container in ECS
In your task definition, update the options for your FireLens configuration:
{
"containerDefinitions": [
{
"essential": true,
"image": "aws_account_id.dkr.ecr.region.amazonaws.com/custom-fluent-bit:latest",
"name": "log_router",
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"config-file-type": "file",
"config-file-value": "/logDestinations.conf"
}
}
}
]
}
For more details ECS FluentBit task definition with an example see AWS Doc here