Multi-Availability Zone (AZ) setup for Kfuse-Pipeline

Our platform supports deploying pods across multiple availability zones to ensure high availability and fault tolerance. Each zone will have at least one pod running for supported components.

Currently, this multi-AZ support is available for the following components:

Ingress-NGINX
Redis (Master-Slave setup)
Ingester

Prerequisites:

Ensure that you are using kfuse-pipeline-3.0 or a later version.

This setup helps maintain service continuity even in the event of a zone-level failure by distributing workloads across zones.

Configuration for multi-AZ cluster setup that needs to be done in custom-values.yaml file:

ingester:
# For multi-AZ cluster setup, replica counts counts should be k * (number of different zones in the cluster) where k= 1,2,3... 
  replicaCount: 1
  service:
    annotations:
      service.kubernetes.io/topology-mode: Auto   

ingress-nginx:
  controller:
    service:
      annotations:
        service.kubernetes.io/topology-mode: Auto
    # replicaCount should be equal to k * (number of different zones in the cluster) where k= 1,2,3... for multi-AZ setup
    replicaCount: 3
    #   This is needed to allow restarts with strict podAntiAffinity (one pod can be destroyed so that a new one is created on the same node)
    updateStrategy:
      rollingUpdate:
        maxUnavailable: 1
      type: RollingUpdate
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/component: controller
              # This should be the same as the release name
              app.kubernetes.io/instance: kfuse-pipeline
          topologyKey: "kubernetes.io/zone"

redis:
  replica:
    # replicaCount - Specify the number of postgres replicas to deploy
    replicaCount: 1     # For multi-AZ cluster setup, replica counts counts should be k * (number of different zones in the cluster) where k= 1,2,3... , otherwise 1
    updateStrategy:
      rollingUpdate:
        maxUnavailable: 1
      type: RollingUpdate
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: kfuse-redis
              app.kubernetes.io/component: replica
              # This should be the same as the release name
              app.kubernetes.io/instance: kfuse-pipeline
          topologyKey: "kubernetes.io/zone"
  sentinel:
    service:
      annotations:
        service.kubernetes.io/topology-mode: Auto