Versions Compared

Version Old Version 2 New Version 3
Changes made by

Suryakant Kashyap

Suryakant Kashyap

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Setup Kfuse-profiler agent to scrape the profiling data

Info

Prerequisite:

1. Ensure your Golang application exposes pprof endpoints.

2. In pull mode, the collector, Alloy, periodically retrieves profiles from Golang applications, specifically targeting the /debug/pprof/* endpoints.

3. If your go code is not setup to generate profiles, you need to setup Golang profiling as mentioned here (for Go Pull mode).

For java, follow the instructions here to setup profiling.

4. Alloy then queries the pprof endpoints of your Golang application, collects the profiles, and forwards them to the Kfuse Profiler server.

...

1. pyroscope.write

2. pyroscope.scrape

1.

...

Configure pyroscope.write

...

block

The pyroscope.write block is used to define the endpoint where profiling data will be sent.

  1. Change url to https://<KFUSE ENDPOINT/DNS NAME>/profile.

  2. Change write_job_name to appropriate name like kfuse_profiler_write.

Code Block
pyroscope.write "write_job_name" {
    endpoint {
        url = "httphttps://localhost:4040<KFUSE ENDPOINT/DNS NAME>/profile"
    }
}

  • Replace "write_job_name" with a unique name for the write job.

  • Update the url with the appropriate endpoint for your Pyroscope server.

2. Add pyroscope.scrape Block

...

2. Configure pyroscope.scrape block

The pyroscope.scrape block is used to define the scraping configuration for profiling data.

  1. Change scrape_job_name to appropriate name like kfuse_profiler_scrape.

  2. Use discovery.relabel.kubernetes_pods.output as a target for pyroscope.scrape block to discover kubernetes targets. Follow steps here to setup specific regex rules for discovering kubernetes targets.

Code Block
pyroscope.scrape "scrape_job_name" {
        targets    = [{"__address__" = "localhost:4040", "service_name" = "example_service"}]
   concat(discovery.relabel.kubernetes_pods.output)
         forward_to = [pyroscope.write.write_job_name.receiver]

        profiling_config {
                profile.process_cpu {
                        enabled = true
                }

                profile.godeltaprof_memory {
                        enabled = true
                }

                profile.memory { // disable memory, use godeltaprof_memory instead
                        enabled = false
                }

                profile.godeltaprof_mutex {
                        enabled = true
                }

                profile.mutex { // disable mutex, use godeltaprof_mutex instead
                        enabled = false
                }

                profile.godeltaprof_block {
                        enabled = true
                }

                profile.block { // disable block, use godeltaprof_block instead
                        enabled = false
                }

                profile.goroutine {
                        enabled = true
                }
        }
}

  • Replace "scrape_job_name" with a unique name for the scrape job.

  • Update the targets field with the appropriate service address and name.

Configuration Details

pyroscope.write:

...

Defines where profiling data should be written.

...

}
        }
}

Configuration Details

  • pyroscope.scrape:

    • Specifies the targets to scrape profiling data from.

    • The forward_to field connects the scrape job to the write job.

    • The profiling_config block enables or disables specific profiles:

      • profile.process_cpu: Enables CPU profiling.

      • profile.godeltaprof_memory: Enables delta memory profiling.

      • profile.memory: Disabled to avoid redundancy with godeltaprof_memory.profile.godeltaprof_mutex: Enables delta mutex profiling.profile.mutex: Disabled to avoid redundancy with godeltaprof_mutex.

      • profile.godeltaprof_block: Enables delta block profiling.

      • profile.block: Disabled to avoid redundancy with godeltaprof_block.profile.goroutine: Enables goroutine profiling.

...

After adding the above blocks to the Alloy configuration file, save the changes and install alloy.

Code Block