Converting old APM data to Kfuse 2.6.5 APM Service Identity format

Service Identity is introduced in Kfuse version 2.6.5 for APM stream. APM data ingested prior to 2.6.5 is incompatible and thus will not get rendered properly in the APM UI page. As part of Kfuse version 2.6.5 release, it includes an option to convert the older data to the current format. Note that the conversion process can take some time, depending on the size of data to convert. When enabled the conversion will run when pinot servers starts up and loads the segments.

Pre-conversion:

To enable the conversion, ensure that the custom_values.yaml has the following configuration:

pinot:
  traces:
    serviceHashConversionEnabled: true
  traces_errors:
    serviceHashConversionEnabled: true
  metrics:
    serviceHashConversionEnabled: true

Disable KV Cardinality limit on Pinot Metrics Table

pinot:
  metrics:
    kvTotalCardinalityThreshold: 0

Increase heap allocated to Pinot Server Offline servers

Segment conversion requires memory. Temporarily double the memory configured for the Pinot server offline in custom_values.yaml.

pinot:
  server:
    offline:
      jvmOpts: "<Adjust the Xmx and Xms settings here>"

Reduce the helix threads to 10
kubectl port-forward -n kfuse pinot-controller-0 9000:9000 curl -X POST "http://localhost:9000/cluster/configs" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"STATE_TRANSITION.maxThreads\": \"10\"}" # Verify using: curl GET "http://localhost:9000/cluster/configs"

Upgrade

Run the standard upgrade command with the updated custom_values.yaml. Refer to Upgrade | Upgrade command

Post-Upgrade

The upgrade includes changes to Pinot table configuration. Restart Pinot servers to ensure that the configuration gets picked up.
kubectl rollout restart sts -n kfuse pinot-server-offline pinot-server-realtime

Conversion Status

Conversion of all Pinot segments can take some time to complete. From the Pinot controller UI console, the table segments status should reflect the loaded (converted) segments. Segments that are in good status are those that have been converted. Once all segments are in good state, then conversion has finished. The command below shows how to connect to Pinot controller.
# Create port-forward to the pinot controller kubectl port-forward -n kfuse pinot-controller-0 9000:9000 # From the browser, go to localhost:9000

Post-Conversion

After conversion has completed, revert the helix threads back to the default setting.
kubectl port-forward -n kfuse pinot-controller-0 9000:9000 curl -X DELETE "http://localhost:9000/cluster/configs/STATE_TRANSITION.maxThreads" -H "accept: application/json"
Revert back the cardinality threshold configuration and heap allocated to the Pinot server offline servers in custom_values.yaml
Run the upgrade command again

Special Cases

To force a re-conversion of segments before the upgrade, delete the pinot-server-offline STS and PVC, and run the conversion steps. This will force older segments to be downloaded from deep store.
kubectl delete sts -n kfuse pinot-server-offline kubectl delete pvc -l component=server-offline -n kfuse