Content Comparison

...

kubectl delete deployments.apps catalog-service rulemanager advance-functions-service

Post-Upgrade

Restart Pinot Services

kubectl rollout restart sts pinot-broker pinot-controller pinot-server-realtime pinot-server-offline

Since we moved hydration-service (HS) from a deployment to statefulset, post upgrade we need to manually delete the pod associated with it.

...

1. Set pinot.server.realtime.replicaCount to 0. Keep note of the original value of this field because we will set it back to that value in step 4.

2. Run helm upgrade as usual.

3. Make sure all pods and jobs have finished successfully.

4. Set pinot.server.realtime.replicaCount back to its original value in the values.yaml file.

5. Either run helm upgrade again.

6. Alternatively, you could run kubectl scale sts pinot-server-realtime --replicas=N

...

Post-Upgrade

1. Connect to rbacdb

> ./kfuse-postgres.sh kfuse-configdb-0 kfuse rbacdb 

2. Make a note of user_id with NULL value created during rbac migration

rbacdb=# select id from users where grafana_id=NULL;

3. Cleanup of empty users in the rbac db created during import from existing users.

rbacdb=# delete from users where grafana_id=NULL;

• For each user in the output of step 2, delete the user from group

rbacdb=# delete from group_members where user_id='<user-id>';

The kfuse-postgres.sh script is available in the customer repository under scripts directory

...

• SLO is reenabled in 2.6.7 with enhanced features.

> ./kfuse-postgres.sh kfuse-configdb-0 kfuse slodb 

slodb=# drop table slodbs;

The kfuse-postgres.sh script is available in the customer repository under scripts directory

...

• Based on observation and feedback, it seems the current persistent volume size for zookeeper pods is getting full quite often. To remediate that we have increased the default size of all zookeeper pods to 32Gi. It needs changes in two places as shown below

kafka:
  # zookeeper - Configuration for Kafka's Zookeeper.
  zookeeper:
    persistence:
      size: 32Gi
      .
      .
      .
pinot:
  # zookeeper - Configuration for Pinot's Zookeeper.
  zookeeper:
    persistence:
      size: 32Gi

• Please update this prior to update with the resize_pvc.sh script. Please reach out if you need assistance.

...

• There are few changes in the pinot database which requires the some of the services to be restarted post upgrade with following command

kubectl rollout restart sts -n kfuse pinot-server-offline pinot-server-realtime pinot-controller pinot-broker logs-parser logs-query-service
kubectl rollout restart deployment -n kfuse logs-transformer trace-transformer trace-query-service

Upgrading to Kfuse version 2.5.0

...

• There are few changes in the pinot database which requires the some of the services to be restarted post upgrade with following command

kubectl rollout restart sts -n kfuse pinot-server-offline pinot-server-realtime pinot-controller pinot-broker logs-parser logs-query-service
kubectl rollout restart deployment -n kfuse logs-transformer

Upgrading to Kfuse version 2.2.4

...

• There are few changes in the pinot database which requires the pinot-* servers to be restarted post upgrade with following command

kubectl rollout restart sts -n kfuse pinot-server-offline pinot-server-realtime pinot-controller pinot-broker

Upgrading to Kfuse version 2.2.3

...

• The default value for pinot zookeeper persistence (PVC) value is now 32Gi.

• If the existing Kfuse installation is using the default value (i.e., custom_values.yaml did not explicitly specify the persistence size for pinot zookeeper), update the pinot zookeeper persistence (PVC) value to 16Gi. Add the following snippet under pinot.zookeeper section:

persistence:
  size: 16Gi

Upgrading from Kfuse version 2.1 or earlier

...

• Kloudfuse provided alerts organization has been updated for better maintenance. Make sure to remove old version.

# Connect to kfuse cluster and log in to catalog service pod
kubens kfuse
kubectl exec -it catalog-servicexxx -- bash
# Remove older folders.
python3 /catalog_service/catalog.py --remove_installed --list kloudfuse,kloudfuse_alerts,kubernetes_alerts --artifact_type alerts

Upgrading from Kfuse version 2.0.1 or earlier

...

• The Kfuse-provisioned dashboard has been cleaned up. Run the following command:

kubectl -n kfuse exec -it kfuse-configdb-0 -- bash -c "PGDATABASE=alertsdb PGPASSWORD=\$POSTGRES_PASSWORD psql -U postgres -c 'delete from dashboard_provisioning where name='\''hawkeye-outliers-resources'\'';'; "

Upgrading from Kfuse version 1.3.4 or earlier

...

• Older kubernetes secret related configuration needs to be removed from the custom values.yaml file. Also kfuse-credentials secret can be removed.

auth:
  config:
    AUTH_TYPE: "google"
    AUTH_COOKIE_MAX_AGE_IN_SECONDS: 259200
  auth:
    existingAdminSecret: "kfuse-credentials"
    existingSecret: "kfuse-credentials"

• There is a schema change introduced in the traces table. Make sure to restart the Pinot servers after upgrade completes.

kubectl rollout restart sts -n kfuse pinot-server-realtime
kubectl rollout restart sts -n kfuse pinot-server-offline

Upgrading from Kfuse version 1.3.2 or earlier

...

• There is a schema change introduced in version 1.3.3. Make sure to restart the Pinot servers after upgrade completes.

kubectl rollout restart sts -n kfuse pinot-server-realtime
kubectl rollout restart sts -n kfuse pinot-server-offline

Upgrading from Kfuse version 1.2.1 or earlier

...

• Advance monitoring is available as an optional component in 1.3 and later releases. To enable:

  • Knight agent is required to be installed. Please review steps/settings here.

  • Additional agent settings required. Please review settings here.

• Starting Kfuse version 1.3.0, Kfuse has added retention support using Pinot Minion framework. This feature requires changes to the existing Pinot Minion statefulset. The statefulset needs to be deleted prior to upgrade.

kubectl delete sts -n kfuse pinot-minion

• Existing alerts have to be updated with new, efficient version. Please follow these steps to refresh the alerts.

  • Go to Kloudfuse UI → Alerts → Alert Rules

  • Using the filter panel on the left, expand “Component” item, and filter all the “Kloudfuse” and “Kubernetes” alerts to include only these. Remove each of these alerts from the filtered list. (post upgrade the new alerts will be installed automatically)

...