GCP Logs Collection Using Pub/Sub

Setup gcp subscription

  • Go to the Cloud Pub-Sub console and create a new topic.

  • Go back to the Pub/Sub Topics overview page, and add select Subscriptions in the left-hand navigation. Select Create Subscription.

  • Create a subscription ID kloudfuse-gcp-subscription and select the topic you previously created.

  • Go to the Logs Explorer page and click on “More actions”-> ”Create sink”

  • Provide a name for the sink.

  • Choose Cloud Pub/Sub as the destination and select the pub/sub that was created for that purpose.

  • Choose what logs to include/exclude from the sink.

  • Click Create Sink and wait for the confirmation message to show up.

Configure Kloudfuse to consume from gcp subscription

Add the following configuration to values.yaml while installing kloudfuse using helm and fill in the GCP pub/sub access key in the pubsubkey field

logs-parser: gcpConfig: enabled: true subscriptionId: "kloudfuse-gcp-subscription" pubsubKey: ""

 

Related pages