Ingest logs from AWS S3 bucket using Filebeat

You can use the filebeat agent and configure it to ingest files in a s3 bucket and ingest logs in Kloudfuse. Use the https://kloudfuse.atlassian.net/wiki/x/CYAEL documentation to setup the the Filebeat and configure the s3 bucket details as below,

daemonset:
  enabled: false
deployment:
  enabled: true
  filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:
      - type: aws-s3
        number_of_workers: 5
        bucket_arn: '<<TBA s3 bucket ARN>>'
        access_key_id: '<<TBA>>' # Not needed in case of Node IAM Role
        secret_access_key: '<<TBA>>' # Not needed in case of Node IAM Role
        endpoint: amazonaws.com
        default_region: "${AWS_REGION:<<TBA>>}"
        fields:
          source: <<TBA Define the source for import> # The logs will be visible with the same source.
      setup.ilm.enabled: false
      setup.template.enabled: false
      output.elasticsearch:
        hosts: ["https://<<TBA Kloudfuse Endpoint>>:443/ingester/api/v1/filebeat"]

If you are ingesting from S3 bucket with Node IAM role, you need to have following permissions added to the policy configured

				"s3:ListAllMyBuckets",
				"s3:GetBucketLocation",
				"s3:ListBucket",
				"s3:GetObject"

Once configured, the Filebeat will import the logs in the Kloudfuse endpoint configured.

Please reach out to the Kloudfuse team in case of any queries in the configuration.

External

Ingest logs from AWS S3 bucket using Filebeat

Related content