Upgrade validation
- Vinay Krishnamurthy
- Abhishek Chaturvedi (Unlicensed)
Verify alerts are updated
If pre/post upgrades were successfully executed, backend should not be getting queries with 1 second step. Verify this by looking at the query-service logs (grep for step:[1] and there shouldn’t be any)
Verify deepstore settings
During an upgrade it’s extremely important that deepstore settings are retained, and that we don’t accidentally remove the deepstore configuration. In order to verify that, we can run helm diff plugin to check the differences.
As a prerequisite install the helm diff plugin by running:
helm plugin install https://github.com/databus23/helm-diffYou can build this plugin from the source and install it too. You can find instructions for building from source here.
Run the following command to generate the diff with the new version:
helm diff upgrade kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse --version <kfuse-version> --values values.yaml,[aws|gcp].yaml[,custom_values.yaml] > /tmp/kfuse_upgrade_diffEnsure that you’ve run docker login before running the diff command. To run the correct command for login, please refer to this section in the Installation page.
If you’re running your cluster on GCP cluster and have a custom values file called custom_values.yaml and upgrading to kfuse version 1.0.5, then you can the above command as:
helm diff upgrade kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse --version 1.0.5 --values values.yaml,gcp.yamlcustom_values.yaml > /tmp/kfuse_upgrade_diffIf the deepstore settings have changed, then you’ll changes in pinot-controller.conf and pinot-server.conf (search or grep for them in /tmp/kfuse_upgrade_diff file):
This sample output shows that the deepstore settings will change from enabled to not enabled with the upgrade. In that case, double-check your deepstore settings in your custom values file.
If you’re unsure about where to find the deepstore settings and how to fix it, reach out to us on our slack channel or email us. Check out this page on how to reach us.