Using Kfuse across different AWS accounts (or VPCs) using PrivateLink and VPC Service Endpoints
- Abhishek Chaturvedi (Unlicensed)
- Pankaj Thakkar
Following network diagram describes the deployment architecture for Kloudfuse when using PrivateLink and VPC Endpoint in AWS.
Steps:
Setup Secure ingestion with private DNS and VPC Endpoint using PrivateLink as described below.
Secure Ingestion using private DNS and VPC Endpoint
Customer Account 2 Configuration
Connect to kubernetes cluster where kfuse stack is installed, list the network load balancer created by the stack. Use following command. It should output a load balancer name of the format
a081addcedc4d299cfa6asdf2143bsdf9-7fe3asf1be2easd.elb.us-west-2.amazonaws.com
kubens kfuse
kubectl get service | grep ingress | grep internal | awk -F' ' '{print $4}'In AWS console on the VPC dashboard select Endpoint Services then Create Endpoint Service. Select the load balancer from step 1 for the endpoint service target.
Wait for the VPC endpoint to have an Available status. Select it, go to Actions, then select Add principals to whitelist.
Here we can add the ARN of the account to which we want to provide access, using the format
arn:aws:iam::<Customer-Account-1-ID>:root. Once you’ve entered the value, select Add to Whitelisted principals.Go to the VPC endpoint service details page and copy the Service name (which will look like:
com.amazonaws.vpce.us-west-2.vpce-svc-asasdf96asdfsadf6) which we'll need later on.
Customer Account 1 Configuration
Create a VPC endpoint in the consumer account 1. Go the VPC dashboard and select Endpoints then Create Endpoint.
under Service category choose Other endpoint services
In Service settings enter the service name of the VPC endpoint service you created in the previous step (
com.amazonaws.vpce.us-west-2.vpce-svc-asasdf96asdfsadf6)select Verify to validate the service name
under VPC select the VPC of account 1 you want to connect
under Security group select security group associated with VPC of account 1.
Select Create endpoint
The VPC endpoint should have a pending acceptance status, which means we need to accept the request in account 2. Go to account 2’s VPC dashboard, go to Endpoint Services, select the endpoint service, then select Endpoint Connections where you should see the pending connection. Select it, then go to Actions and choose Accept endpoint connection request.
Confirm the acceptance on the popup that appears, then wait for the endpoint to move from Pending to Available
Back in Account 1, under your endpoint details there should be a list of several DNS names. Copy the first one, which allows you to connect to the VPC endpoint from any availability zone. It would be of the form:
vpce-sasdgsadfasdf-01z8ed5t.vpce-svc-asasdf96asdfsadf6.us-west-2.vpce.amazonaws.com.Update
dd-values-kfuse.yamlfile by replacingdd_urlfor various streams as follows using the DNS name from step 5:
datadog:
dd_url: "http://vpce-sasdgsadfasdf-01z8ed5t.vpce-svc-asasdf96asdfsadf6.us-west-2.vpce.amazonaws.com/ingester"
...
process_config:
process_dd_url: "http://vpce-sasdgsadfasdf-01z8ed5t.vpce-svc-asasdf96asdfsadf6.us-west-2.vpce.amazonaws.com/ingester"
events_dd_url: "http://vpce-sasdgsadfasdf-01z8ed5t.vpce-svc-asasdf96asdfsadf6.us-west-2.vpce.amazonaws.com/ingester"
...
orchestrator_explorer:
orchestrator_dd_url: "http://vpce-sasdgsadfasdf-01z8ed5t.vpce-svc-asasdf96asdfsadf6.us-west-2.vpce.amazonaws.com/"
...
logs_config:
logs_dd_url: "vpce-05ea299fea1593ce3-01z8ed5t.vpce-svc-05ccc49663c67653f.us-west-2.vpce.amazonaws.com:80" Reinstall agent with updated values.