Ingest logs from s3 bucket using Filebeat

You can use the filebeat agent and configure it to ingest files in a s3 bucket and ingest logs in Kloudfuse. Use the https://kloudfuse.atlassian.net/wiki/x/CYAEL documentation to setup the the Filebeat and configure the s3 bucket details as below,

daemonset:
  enabled: false
deployment:
  enabled: true
  filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:
      - type: aws-s3
        number_of_workers: 5
        bucket_arn: '<<TBA s3 bucket ARN>>'
        access_key_id: '<<TBA>>'
        secret_access_key: '<<TBA>>'
        endpoint: amazonaws.com
        default_region: "${AWS_REGION:<<TBA>>}"
        fields:
          source: <<TBA Define the source for import> # The logs will be visible with the same source.
      setup.ilm.enabled: false
      setup.template.enabled: false
      output.elasticsearch:
        hosts: ["https://<<TBA Kloudfuse Endpoint>>:443/ingester/api/v1/filebeat"]

Once configured, the Filebeat will import the logs in the Kloudfuse endpoint configured.

Please reach out to the Kloudfuse team in case of any queries in the configuration.