HTTPS/TLS Setup on Kloudfuse Stack

Owned by Harold Lim
Last updated: Aug 28, 2024
8 min read

The Kloudfuse helm chart supports 2 options for setting up HTTPS/TLS on the Kloudfuse Stack: 1. Configuring HTTPS/TLS on the Kubernetes Ingress rules. 2. TLS termination before traffic hits the Kloudfuse stack.

 

Note that, if TLS is enabled, the Grafana settings for alerts need to be updated. Instructions is at the end of this document.

Instructions for Setting HTTPS/TLS on the Kloudfuse Ingress

Note. If using AWS Certificate Manager (ACM) to terminate the tls on the AWS NLB, skip to the next section for instructions.

This section describes setting up HTTPS/TLS on the Ingress rules. The Kloudfuse helm chart supports creation of the certificates, using cert-manager and lets-encrypt or use an existing certificate.

Prerequisites

  1. A DNS host name is required.

  2. A static external IP address associated with the DNS host name that can be used by the ingress load balancer.

    1. For GKE, refer to https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address for reserving a static external IP.

    2. For AWS, refer to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html for allocating IP addresses from AWS Elastic IP.

Step 3.a. is only needed if SSL certificate will be created as part of Kfuse installation. Certificate will be created by the cert-manager using lets-encrypt.

3. a. cert-manager must already be installed in the kubernetes cluster. Refer to https://cert-manager.io/docs/installation/ or follow the instructions below.

image-20240612-144933.png

 

helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager --create-namespace --namespace kfuse --version v1.7.1 --set installCRDs=true

 

Step 3.b. is needed if existing SSL certificate will be used.

3.b. Create a Kubernetes secret of type TLS with the server.crt and server.key files (SSL certificates).

 

image-20240612-145258.png

 

kubectl create secret tls kfuse-tls --namespace kfuse --key server.key --cert server.crt

Helm Values Configuration

Add the following entries in the custom_values.yaml depending on the cloud provider.

GCP/Azure

tls: enabled: true host: <REPLACE DNS HOST HERE> email: <REPLACE EMAIL> # If Step 3.a. clusterIssuer: <PROVIDE A CLUSTER ISSUER NAME. ensure that this is unique within the cluster, leave blank if providing own ssl certificate> # If Step 3.b. secretName: <PROVIDE secret name containing SSL certificate, e.g., kfuse-tls in Step 3.b. example command> ingress-nginx: controller: ingressClassResource: enabled: true name: kfuse-ingress ingressClass: kfuse-ingress watchIngressWithoutClass: false service: loadBalancerIP: <REPLACE PRE-ALLOCATED STATIC IP HERE> external: enabled: true

AWS

For AWS, instead of directly specifying the IP address, in the ingress loadbalancerIP, assignment is done through
service annotations.

Note that the number of Elastic IP allocations must match the number of subnets the AWS NLB uses (defaults to 3).
Refer to https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html and
https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/service/annotations/

Note: on some older cluster of EKS, the following annotation:

may need to be replaced with

Disable HTTP traffic

If you want to disable HTTP traffic completely, add the following to ingress-nginxsection in values.yaml

Note that this will disable HTTP traffic on both internal and external ingress. So use this config if you want to disable HTTP, and you’re not using internal ingress.

Instructions to Configure Kloudfuse stack to work with TLS Termination

The Kloudfuse stack can be configured to work on clusters that have TLS termination (e.g., using OKTA, AWS ACM).

Minimal instructions

Note: Skip to the next section for more detailed instructions if using AWS Certificate Manager and AWS NLB for TLS termination.

At the minimum, add the following entries in the custom_values.yaml.

 

The third-party service that does the TLS termination will need to be updated to point to the external ip of the kfuse-ingress-controller service.

Instructions to configure TLS termination on AWS NLB using AWS Certificate Manager

AWS supports configuring the TLS termination directly on the AWS NLB created that backs the Kfuse ingress service.

Prerequisites

  1. A DNS host name is required.

  2. A certificate associated with the above DNS host name issued by the AWS Certificate Manager.

    1. Refer to https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html

  3. Elastic IP(s) associated with the DNS host name that can be used by the ingress load balancer.

    1. Refer to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html for allocating IP addresses from AWS Elastic IP.

Helm Values Configuration

Following are the example configurations for nginx-ingress with their use-cases

Minimal configuration of nginx-ingress controller with ACM

 

Configuration of nginx-ingress controller, enabling host based routing (Only External Load Balancer)

 

 

This will restrict the Kfuse access only to the host specified in the configuration.

Configuration of nginx-ingress controller, enabling host based routing (Both Internal and External Load Balancer)

 

 

 

This will restrict the Kfuse access only to the host (internal & external) specified in the configuration. In this configuration the the internal load balancer can only be used for ingest path i.e. ingesting MELT data and query path is only available via external load balancer.

Configure internal load balancer for sending agent data on internal network

Ensure that the certificate created for external load balancer has been updated to contain the domain-name for the internal load balancer. Update the DNS entries as required. ACM only allows immediate subdomains (*.domain.com) or sibling domains (internal.domain.com). For example; if the kfuse stack is hosted at observability.domain.com; internal traffic can be routed to internal.observability.domain.com; For this the certificate can contain the two domain names. Once the ACM certificate has been resolved; update the custom_values.yaml ingress section for internal controller.