{"type":"doc","content":[{"type":"paragraph","content":[{"text":"Kloudfuse is installed on a kubernetes cluster (gke/eks, etc.) by a single helm chart. The data is stored in persistent volumes and backed up on cloud storage (GCS/S3, etc.). The backup and recovery for region/availability zone failures will be handled through the respective cloud provider’s kubernetes cluster backup and recovery feature.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Google Cloud Backup and Recovery for GKE","type":"text"}]},{"type":"paragraph","content":[{"text":"To handle region/availability zone failures, we recommend the following steps:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create a GKE cluster as per the installation docs in the primary region/availability zone","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create another GKE cluster as per the installation docs in the failover region/availability zone","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For DNS/TLS setup, run the pre-requisite steps on both the GKE clusters as per the ","type":"text"},{"text":"instructions here","type":"text","marks":[{"type":"link","attrs":{"href":"https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/789774337"}}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Enable GKE backup and recovery for the above clusters as per the ","type":"text"},{"text":"GKE instructions","type":"text","marks":[{"type":"link","attrs":{"href":"https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/how-to/backup-plan#before_you_begin"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create GCS bucket with cross region/availability zone access as needed","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install kloudfuse helm chart in the primary gke cluster - see the ","type":"text"},{"text":"installation instructions","type":"text","marks":[{"type":"link","attrs":{"href":"https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/724238341"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For any secrets and config maps created by the user (e.g. for SSO/SAML setup, TLS certs, etc.), following labels to be added: ","type":"text"}]},{"type":"codeBlock","content":[{"text":"app.kubernetes.io/instance: kfuse","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Setup GKE backup policy based on the RPO and retention needed. This will automatically create backups of kloudfuse installations to meet the selected RPO. The backups will be retained for the given period and deleted afterwards. ","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"gcloud beta container backup-restore backup-plans create kloudfuse-backup-plan \\\n --project= \\\n --location= \\\n --cluster= \\\n --selected-applications=/kloudfuse \\\n --include-secrets \\\n --target-rpo-minutes=60 \\\n --backup-retain-days=1","type":"text"}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Note: The minimum RPO that can be set is 60 minutes using the option ","type":"text"},{"text":"--target-rpo-minutes","type":"text","marks":[{"type":"code"}]},{"text":". If you want RPO to be lesser than 60 minutes, use the option ","type":"text"},{"text":"--cron-schedule.","type":"text","marks":[{"type":"code"}]},{"text":" Also, note that these two options cannot be used together in one backup plan: ","type":"text"},{"text":"--target-rpo-minutes","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"--cron-schedule","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For setting up GKE backup policy based on ","type":"text"},{"text":"--cron-schedule","type":"text","marks":[{"type":"code"}]},{"text":". For example ","type":"text"},{"text":"\"10 3 * * *\"","type":"text","marks":[{"type":"code"}]},{"text":" creates a backup at 3:10 AM every day. All times are interpreted as UTC. ","type":"text"}]},{"type":"codeBlock","content":[{"text":"gcloud beta container backup-restore backup-plans create kloudfuse-backup-plan \\\n --project= \\\n --location= \\\n --cluster= \\\n --selected-applications=/kloudfuse \\\n --include-secrets \\\n --cron-schedule=\"10 3 * * *\" \\\n --backup-retain-days=1","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Setup GKE restore plan","type":"text"}]},{"type":"codeBlock","content":[{"text":"gcloud beta container backup-restore restore-plans create kloudfuse-restore-plan \\\n --project= \\\n --location= \\\n --backup-plan=kloudfuse-backup-plan \\\n --cluster= \\\n --cluster-resource-conflict-policy=delete-and-restore \\\n --selected-applications=/kloudfuse \\\n --volume-data-restore-policy=restore-volume-data-from-backup","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"On failure of the primary region/availability zone (this needs to be detected out of band), administrator will restore the kloudfuse to the target/failover GKE cluster.","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"gcloud beta container backup-restore restores create kloudfuse-restore \\\n --project= \\\n --location= \\\n --restore-plan=kloudfuse-restore-plan \\\n --backup=

Browser not supported