Upgrade

Upgrade command

Before performing an upgrade, validate that the upgrade won’t revert any customization on your cluster. The steps to run a validation are detailed in here.

To check the current installed Kfuse version, run the following command:

The upgrade command is the same as the install command specified in https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/724238341. The command is copied here for reference. The latest released version is specified in the installation page.

Upgrading to Kfuse version 2.6.7

Note that Identity for Databases is introduced in Kfuse version 2.6.7. Database Identity only takes effect on new ingested APM-related data. In addition, timestamp granularities for APM/span data has been increased from millisecond to nanosecond to provide better accuracy in Trace Flamegraph/Waterfall. For older APM data to be rendered accurately, follow the instructions https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/984416304 to convert old data to the new format.

Pre-Upgrade

• SLO is reenabled in 2.6.7 with enhanced features. If there is no SLO setup, then this step can be skipped.

The kfuse-postgres.sh can be obtained through Kloudfuse support.

Post-Upgrade

• There are few changes in the pinot database which requires Pinot server to be restarted post upgrade with following command

Upgrading to Kfuse version 2.6.5

Note that Service Identity for APM is introduced in Kfuse version 2.6.5. Service Identity only takes effect on new ingested APM-related data. Accordingly, old APM data will not get rendered properly in the UI. If older APM data is needed. Then follow the instructions https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/984416304 to convert old data to the new format.

Pre-Upgrade

• On Azure, The kfuse-ssd-offline storage class is changed toStandardSSD_LRS disk type. The kfuse-ssd-offline storage class needs to be deleted prior to upgrade to allow the new version to update the disk type. Note that if the installation is not on Azure, then this step can be skipped.

Post-Upgrade

• There are few changes in the pinot database which requires Pinot server to be restarted post upgrade with following command

Upgrading to Kfuse version 2.6

Pre-Upgrade

• A new kfuse-ssd-offline storage class has been introduced in Kfuse version 2.6. This storage class uses gp3 on AWS, pd-balanced on GCP, and Standard_LRS on Azure. This is now the default storage class for Pinot Offline Servers, which should give better disk IO performance.

• If the custom values yaml is already set to use the specified disk type (e.g., kfuse-ssd-aws-gp3 or standard-rwo on GCP), then the remaining steps can be skipped.

• If the custom values yaml does not explicitly set the pinot.server.offline.persistence.storageClass field or it is set to a different storage class. Ensure that the field is not set in the custom values yaml. Then run the following commands

• Note that the above commands corresponds to the PVCs of the Pinot offline servers. After upgrade to Kfuse version 2.6, PVCs with the desired storage class will be created for the Pinot offline servers.

Upgrading to Kfuse version 2.5.3

Pre-Upgrade

• Based on observation and feedback, it seems the current persistent volume size for zookeeper pods is getting full quite often. To remediate that we have increased the default size of all zookeeper pods to 32Gi. It needs changes in two places as shown below

• Please update this prior to update with the resize_pvc.sh script. Please reach out if you need assistance.

Post-Upgrade

• There are few changes in the pinot database which requires the some of the services to be restarted post upgrade with following command

Upgrading to Kfuse version 2.5.0

Post-Upgrade

• There are few changes in the pinot database which requires the some of the services to be restarted post upgrade with following command

Upgrading to Kfuse version 2.2.4

Post-Upgrade

• There are few changes in the pinot database which requires the pinot-* servers to be restarted post upgrade with following command

Upgrading to Kfuse version 2.2.3

Pre-Upgrade

• The default value for pinot zookeeper persistence (PVC) value is now 32Gi.

• If the existing Kfuse installation is using the default value (i.e., custom_values.yaml did not explicitly specify the persistence size for pinot zookeeper), update the pinot zookeeper persistence (PVC) value to 16Gi. Add the following snippet under pinot.zookeeper section:

Upgrading from Kfuse version 2.1 or earlier

Post-Upgrade

• Kloudfuse provided alerts organization has been updated for better maintenance. Make sure to remove old version.

Upgrading from Kfuse version 2.0.1 or earlier

Post-Upgrade

• The Kfuse-provisioned dashboard has been cleaned up. Run the following command:

Upgrading from Kfuse version 1.3.4 or earlier

Pre-Upgrade

• Note: Kfuse services will go offline during this process. Kloudfuse storage class configuration has been simplified to keep future releases/features in mind. This requires running the migrate_storage_class.sh script provided by Kloudfuse team.

  After running the script, ensure that the pvc’s storage class has kfuse-ssd, instead of kfuse-ssd-gcp or kfuse-ssd-aws.

• Old alerts have to be removed. Kloudfuse alerts organization has changed with the introduction of additional alerts. New version does the organization automatically, however, the older alerts have to be removed.

  • Manually remove all alerts by navigating to the grafana tab and remove all alerts from kloudfuse_alerts and kubernetes_alerts folder.

Open

Post-Upgrade

• Older kubernetes secret related configuration needs to be removed from the custom values.yaml file. Also kfuse-credentials secret can be removed.

• There is a schema change introduced in the traces table. Make sure to restart the Pinot servers after upgrade completes.

Upgrading from Kfuse version 1.3.2 or earlier

Post-Upgrade

• There is a schema change introduced in version 1.3.3. Make sure to restart the Pinot servers after upgrade completes.

Upgrading from Kfuse version 1.2.1 or earlier

Pre-Upgrade

• Advance monitoring is available as an optional component in 1.3 and later releases. To enable:

  • Knight agent is required to be installed. Please review steps/settings here.

  • Additional agent settings required. Please review settings here.

• Starting Kfuse version 1.3.0, Kfuse has added retention support using Pinot Minion framework. This feature requires changes to the existing Pinot Minion statefulset. The statefulset needs to be deleted prior to upgrade.

• Existing alerts have to be updated with new, efficient version. Please follow these steps to refresh the alerts.

  • Go to Kloudfuse UI → Alerts → Alert Rules

  • Using the filter panel on the left, expand “Component” item, and filter all the “Kloudfuse” and “Kubernetes” alerts to include only these. Remove each of these alerts from the filtered list. (post upgrade the new alerts will be installed automatically)

Upgrading from Kfuse version 1.1.1 or earlier

Cloud-Specific configurations

Starting Kfuse version 1.2.0, cloud-specific yamls (aws.yaml, gcp.yaml, azure.yaml) are not included in the chart anymore. The custom_values.yaml needs to include these configurations. Refer to and . With Kfuse version 1.2.0, there is no need to pull the chart prior to installation. helm upgrade can be directly run with the Kfuse helm chart registry.

Database changes

A breaking change related to the number of postgresql servers installed as part of the Kfuse install was introduced after Kfuse version 1.1.0. Due to this, the stored alerts will be deleted if directly upgrading Kfuse. In order to retain the stored alerts, run the following pre and post upgrade steps below. Note that until the post-upgrade steps are executed, alerts & dashboards from pre-upgrade will not show up.

Pre-Upgrade

Post-Upgrade

Upgrading from Kfuse version 1.0.4 or earlier

The Kfuse StorageClass resources need to be deleted prior to upgrade.

To check the current installed Kfuse version, run the following command:

To delete the storage class:

Otherwise, to upgrade just follow the install instructions as is.