Using Kfuse across different AWS accounts (or VPCs) using PrivateLink and VPC Service Endpoints

Owned by Abhishek Chaturvedi (Unlicensed)
Last updated: Aug 10, 2023
2 min read

Following network diagram describes the deployment architecture for Kloudfuse when using PrivateLink and VPC Endpoint in AWS.

Steps:

  1. Setup ACM for secure Kfuse UI access.

  2. Setup Secure ingestion with private DNS and VPC Endpoint using PrivateLink as described below.

Secure Ingestion using private DNS and VPC Endpoint

 

Customer Account 2 Configuration

  1. Connect to kubernetes cluster where kfuse stack is installed, list the network load balancer created by the stack. Use following command. It should output a load balancer name of the format a081addcedc4d299cfa6asdf2143bsdf9-7fe3asf1be2easd.elb.us-west-2.amazonaws.com

kubens kfuse kubectl get service | grep ingress | grep internal | awk -F' ' '{print $4}'

  1. In AWS console on the VPC dashboard select Endpoint Services then Create Endpoint Service. Select the load balancer from step 1 for the endpoint service target.

  2. Wait for the VPC endpoint to have an Available status. Select it, go to Actions, then select Add principals to whitelist.

    Here we can add the ARN of the account to which we want to provide access, using the format arn:aws:iam::<Customer-Account-1-ID>:root. Once you’ve entered the value, select Add to Whitelisted principals.

  3. Go to the VPC endpoint service details page and copy the Service name (which will look like: com.amazonaws.vpce.us-west-2.vpce-svc-asasdf96asdfsadf6) which we'll need later on.

Customer Account 1 Configuration

  1. Create a VPC endpoint in the consumer account 1. Go the VPC dashboard and select Endpoints then Create Endpoint.

    1. under Service category choose Other endpoint services

      1. In Service settings enter the service name of the VPC endpoint service you created in the previous step (com.amazonaws.vpce.us-west-2.vpce-svc-asasdf96asdfsadf6)

      2. select Verify to validate the service name

    2. under VPC select the VPC of account 1 you want to connect

      1. under Security group select security group associated with VPC of account 1.

  2. Select Create endpoint

  3. The VPC endpoint should have a pending acceptance status, which means we need to accept the request in account 2. Go to account 2’s VPC dashboard, go to Endpoint Services, select the endpoint service, then select Endpoint Connections where you should see the pending connection. Select it, then go to Actions and choose Accept endpoint connection request.

  4. Confirm the acceptance on the popup that appears, then wait for the endpoint to move from Pending to Available

  5. Back in Account 1, under your endpoint details there should be a list of several DNS names. Copy the first one, which allows you to connect to the VPC endpoint from any availability zone. It would be of the form: vpce-sasdgsadfasdf-01z8ed5t.vpce-svc-asasdf96asdfsadf6.us-west-2.vpce.amazonaws.com.

  6. Update dd-values-kfuse.yaml file by replacing dd_url for various streams as follows using the DNS name from step 5:

datadog: dd_url: "http://vpce-sasdgsadfasdf-01z8ed5t.vpce-svc-asasdf96asdfsadf6.us-west-2.vpce.amazonaws.com/ingester" ... process_config: process_dd_url: "http://vpce-sasdgsadfasdf-01z8ed5t.vpce-svc-asasdf96asdfsadf6.us-west-2.vpce.amazonaws.com/ingester" events_dd_url: "http://vpce-sasdgsadfasdf-01z8ed5t.vpce-svc-asasdf96asdfsadf6.us-west-2.vpce.amazonaws.com/ingester" ... orchestrator_explorer: orchestrator_dd_url: "http://vpce-sasdgsadfasdf-01z8ed5t.vpce-svc-asasdf96asdfsadf6.us-west-2.vpce.amazonaws.com/" ... logs_config: logs_dd_url: "vpce-05ea299fea1593ce3-01z8ed5t.vpce-svc-05ccc49663c67653f.us-west-2.vpce.amazonaws.com:80"

  1. Reinstall agent with updated values.