Ingest logs from AWS S3 bucket using Filebeat

You can use the filebeat agent and configure it to ingest files in a s3 bucket and ingest logs in Kloudfuse. Use the https://kloudfuse.atlassian.net/wiki/x/CYAEL documentation to setup the the Filebeat and configure the s3 bucket details as below,

daemonset:
  enabled: false
deployment:
  enabled: true
  filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:
      - type: aws-s3
        number_of_workers: 5
        bucket_arn: '<<TBA s3 bucket ARN>>'
        access_key_id: '<<TBA>>'
        secret_access_key: '<<TBA>>'
        endpoint: amazonaws.com
        default_region: "${AWS_REGION:<<TBA>>}"
        fields:
          source: <<TBA Define the source for import> # The logs will be visible with the same source.
      setup.ilm.enabled: false
      setup.template.enabled: false
      output.elasticsearch:
        hosts: ["https://<<TBA Kloudfuse Endpoint>>:443/ingester/api/v1/filebeat"]

Once configured, the Filebeat will import the logs in the Kloudfuse endpoint configured.

Please reach out to the Kloudfuse team in case of any queries in the configuration.

External

Ingest logs from AWS S3 bucket using Filebeat

Related pages