Prerequisites

Kloudfuse stack is designed to run in a Kubernetes cluster on GCP, AWS or Azure cloud.

To install Kloudfuse stack you will need:

• Machine with at least 8vCPU and 64GB memory.

• A Kubernetes cluster (separate or can be shared with the application)

• Persistent volumes (these don’t have to manually created. installation will create these, but the account used to install needs to have the necessary permissions for smooth install)

• Helm, kubectl.

• Please note helm version 3.8.1 or above is needed.

Above configuration is good enough to get started. However, for more advanced/production grade customizations please ensure to review the prerequisites for advanced use cases page.

Installation

Kloudfuse follows standard helm best practices. Please follow these steps to install the Kloudfuse stack.

• Step 1. Login to Kloudfuse helm registry using the “token.json" provided in your email. (see here if following command fails). If you do not have this token or haven’t received the email, please contact us.

cat token.json | helm registry login -u _json_key --password-stdin https://us-east1-docker.pkg.dev

• Step 2. Create a namespace called “kfuse” to Install Kloudfuse. Create a secret for helm to pull the required docker images.

  kubectl create ns kfuse kubectl config set-context --current --namespace=kfuse kubectl create secret docker-registry kfuse-image-pull-credentials \ --namespace='kfuse' --docker-server 'us.gcr.io' --docker-username _json_key \ --docker-email 'container-registry@mvp-demo-301906.iam.gserviceaccount.com' \ --docker-password=''"$(cat token.json)"''

• Step 3. create a custom_values.yaml file. If we have shared a yaml file with you then use that as a starting point and modify it as needed.

  • Add an orgId (typically the company name). This is a required field.

    global: orgId: <REPLACE YOUR COMPANY NAME HERE>

  • Add an cloudProvider. This is a required field. Valid values are aws, gcp, or azure

    global: cloudProvider: <aws, gcp, or azure>

  • Optionally, add a section in custom_values.yaml specific to your cloud provider

• Step 4. To install the Kfuse chart, run the following command (Please note helm version 3.8.1 or above is needed):

helm upgrade --install -n kfuse kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse --version 3.1.1  -f custom_values.yaml

UI

Upon completion of the above helm install you would see instructions for accessing the Kloudfuse Platform UI.

Watch the status:

kubectl get svc --namespace kfuse -w kfuse-ingress-nginx-controller
export SERVICE_IP=$(kubectl get svc --namespace kfuse kfuse-ingress-nginx-controller --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
echo http://$SERVICE_IP

Kloudfuse UI and Grafana Access

The Kloudfuse Platform has its own UI for each of the streams and more. You will also see a tab for Grafana if you wish to access your data using Grafana. To access them the default credentials are:

username: admin
password: password

Please follow the steps here to configure Google Auth or change the default password.

Please see next steps to explore more.

Telemetry from Kloudfuse stack

The standard Kloudfuse install sends its telemetry data to itself as well as the Kloudfuse cloud (observe.kloudfuse.io). This is so that we can monitor the health of the installed Kloudfuse stack(s) across our customers and help resolve any issues quickly. If you wish to disable sending telemetry data to Kloudfuse cloud, please contact us .

Next

• Add users or change default admin password

• How to configure google authentication.

• Run kloudfuse stack without an external load balancer.

• Please see Migration scenarios & integrations to understand and choose what suits your need.

More Info

helm registry login failure

• if helm registry login command does not work, you can substitute ‘helm registry login’ with ‘docker login’:

cat token.json | docker login -u _json_key --password-stdin https://us-east1-docker.pkg.dev

Review default values used in the install

• The standard default values of the Kfuse helm chart is configured for a single-node cluster install (without deepstore). The default values of the Kfuse helm chart can be viewed by running the command below. Any additional customization can be added to the custom_values.yaml.

  helm show values oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse --version 2.0.1

Uninstall

To delete the kloudfuse installation, you can helm delete kfuse chart and/or delete the ‘kfuse’ namespace. helm delete does not get rid of the data stored in persistent disk or in the deepstore in the GCP/AWS/Azure cloud buckets.

helm delete kfuse

Deletion of Persistent Disks

kubectl delete pvc -n kfuse -l app.kubernetes.io/instance=kfuse

Deletion of DeepStore Data

GCP:

gsutil -m rm -r gs://<pinot-bucket-name>/controller

AWS:

aws s3 rm s3://<pinot-bucket-name>/controller --recursive