Steps to use Terraform to provision kloudfuse infrasturcture and deploy Kloudfuse

Pre-requisites: For configuring s3 as Terraform backend

S3 bucket - Create an s3 bucket with public access off and with remaining defaults enabled.
Dynamodb table - Create a dynamodb table with partition key = LockID
Install Terraform - Have the latest terraform installed and export the terraform to be part of your executable PATH.
Terraform user - Create a AWS user or use a role that has enough permissions to access the s3 bucket and provision different resources on AWS.
1. Make sure that profile is added to your aws credentials.
2. Export the aws profile before running terraform.

Please note the backend.tf file in each component folder would need to be changed according to the resources created above (s3 and dynamodb) whereas the state key is unique for each component for eg for a vpc it can be vpc.tfstate.

Example:

terraform {
  backend "s3"{
    bucket                 = "bucket-name"
    region                 = "us-west-2"
    key                    = "<resource>.tfstate"
    dynamodb_table         = "dynamodb-table-name"
  }
}

Please note the variables under variables.tf for every aws component need to be modified, refer to the variables table provided below for more information.

Variable name	Example value	Definition

Variable name	Example value	Definition

vpc_cidr	"172.159.0.0/16"	The IPv4 CIDR block for the VPC. CIDR can be explicitly set or it can be derived from IPAM using `ipv4_netmask_length` & `ipv4_ipam_pool_id`
region	“us-west-2”	Region where the resource is being created
azs	["us-west-2a", "us-west-2b", "us-west-2c"]	"A list of availability zones names or ids in the region"
public_subnets	["172.159.0.0/19", "172.159.32.0/19", "172.159.64.0/19"]	A list of public subnets in the region
domain_name	"terraform-dev.kloudfuse.io"	Domain name for the ACM certificate
validation_method	"DNS"	validation method for ACM certificate
cluster_name	"kfuse-devops-eks"	eks cluster name
bucket_deepstore	"kfuse-test-deepstore-s3"	s3 bucket used for deepstore
principal_arn_console	"arn:aws:iam::783739827:user/terraform"	The ARN of the IAM principal (user or role) to grant access to the EKS cluster
principal_arn_cli	"arn:aws:iam::783739827:user/terraform"	The ARN of the IAM principal (user or role) to grant access to the EKS cluster
ami_type	"AL2_x86_64"	ami type to be used
instance_types	["r6i.8xlarge"]	instance type used for eks cluster

If Customer has an AWS account with no infrastructure provisioned

VPC provisioning: Create the VPC using the terraform files provided here.
1. Please note the variables under variables.tf need to be modified according to the customers requirements for eg VPC CIDR, Public subnet CIDR, VPC name etc.
S3 provisioning: Kloudfuse datastore Pinot requires access to a s3 bucket for long term storage. The terraform resources to create the appropriate S3 bucket are under the /terraform/s3 module here.
EKS provisioning: Once the VPC is created, the EKS cluster can be provisioned using the terraform resources under /terraform/eks here
1. Please note the main.tf has a block(terraform_remote_state) that refers to the vpc backend in s3 in order to retrieve values for public subnet ids.
2. Please note the variables under variables.tf need to be modified, refer to the variables table provided here. Importantly, the add the proper ARN for cluster users in the principal variables.
Route53: A public hosted zone is required for the acm certificate to create a record in it. The zone can be created using the terraform files here.
ACM: A certificate is required for ssl/tls enablement. The certificate can be created using the terraform files here.

If Customer has an AWS account with VPC provisioned but is not running an EKS cluster

The components that would be needed are EKS, S3, ACM and Route53. From the above section repeat steps 2-5. Please note for EKS since the VPC was not created using the terraform files provided by Kloudfuse, the public subnet ids would need to be provided in the variables.tf file and used in the main.tf

If Customer has an AWS account with existing EKS cluster (shared or dedicated) and need to deploy kfuse only then there are two ways of installing kfuse outlined below:

Option 1: Terraform + Helm

Important: Before running the next steps make sure to copy the kfuse token.json and your `~/.kube/config(as kubeconfig in destination) files in the /prekfuse folder. Also, the kubeconfig will need to have the config updated for the above created/existing EKS Cluster where Kfuse needs to be deployed.

Run the Kfuse pre-requisite terraform files for setting up the cluster from here, this will create the namespace, login into our helm registry and create a secret required by the services to pull their helm charts.

Run the helm upgrade command as below

helm upgrade --install -n kfuse kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse --version 2.7.4  -f custom_values.yaml

Option 2: Kubectl commands + Helm

Step 1. Login to Kloudfuse helm registry using the “token.json" provided in your email. (see here if following command fails). If you do not have this token or haven’t received the email, please contact us.

cat token.json | helm registry login -u _json_key --password-stdin https://us-east1-docker.pkg.dev

Step 2. Create a namespace called “kfuse” to Install Kloudfuse. Create a secret for helm to pull the required docker images.

Step 3. To install the Kfuse chart, run the following command (Please note helm version 3.8.1 or above is needed)