You can use the filebeat agent and configure it to ingest files in a s3 bucket and ingest logs in Kloudfuse. Use the https://kloudfuse.atlassian.net/wiki/x/CYAEL documentation to setup the the Filebeat and configure the s3 bucket details as below,
daemonset:
enabled: false
deployment:
enabled: true
filebeatConfig:
filebeat.yml: |
filebeat.inputs:
- type: aws-s3
number_of_workers: 5
bucket_arn: '<<TBA s3 bucket ARN>>'
access_key_id: '<<TBA>>'
secret_access_key: '<<TBA>>'
endpoint: amazonaws.com
default_region: "${AWS_REGION:<<TBA>>}"
fields:
source: <<TBA Define the source for import> # The logs will be visible with the same source.
setup.ilm.enabled: false
setup.template.enabled: false
output.elasticsearch:
hosts: ["https://<<TBA Kloudfuse Endpoint>>:443/ingester/api/v1/filebeat"]
Once configured, the Filebeat will import the logs in the Kloudfuse endpoint configured.
Please reach out to the Kloudfuse team in case of any queries in the configuration.