Troubleshooting
Networking
Kfuse is unreachable from external host
Symptom
Unable to access Kfuse from the external IP/host or DNS.
curl http://EXTERNAL_IP
curl: (28) Failed to connect to XX.XX.XX.XX port 80 after 129551 ms: Connection timed out
curl https://EXTERNAL_IP --insecure
curl: (28) Failed to connect to XX.XX.XX.XX port 443 after 129551 ms: Connection timed out
Resolution
Ensure that the security group or firewall policy for the Kubernetes cluster, node, and VPC endpoint allows external incoming traffic.
Packet is getting dropped by ingress-nginx
Symptom
ingress-nginx logs client intended to send too large body
error.
2023/03/06 05:38:22 [error] 43#43: *128072996 client intended to send too large body: 1097442 bytes, client: XXXX, server: _, request: "POST /ingester/v1/fluent_bit HTTP/1.1", host: "XXXX"
Resolution
ingress-nginx can be configured to accept larger request body size. The default is 1m. Upgrade Kfuse with the following section in the custom values file.
ingress-nginx:
controller:
config:
proxy-body-size: <REPLACE THE BODY SIZE HERE, e.g., 8m. Setting to 0 will disable any limit.>
Pinot
DeepStore access issues
Symptoms
Pinot-related jobs are stuck in crash loop back-off (e.g., kfuse-set-tag-hook, pinot-metrics-table-creation, etc).
Pinot-controller logs deep store access-related exception.
On AWS S3, the exception has the following format
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: MAYE68P6SYZMTTMP, Extended Request ID: L7mSpEzHz9gdxZQ8iNM00jKtoXYhkNrUzYntbbGkpFmUF+tQ8zL+fTpjJRlp2MDLNvhaVYCie/Q=)
Resolution
Refer to https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/724664352 for setting the access for Pinot.
On GCP, ensure that the secret has correct access to the cloud storage bucket.
On AWS S3, if the node does not have permission to the S3 bucket, then ensure that the access key and secret access key is populated
pinot: deepStore: enabled: true type: "s3" useSecret: true createSecret: true dataDir: "s3://[REPLACE BUCKET HERE]/kfuse/controller/data" s3: region: "YOUR REGION" accessKey: "YOUR AWS ACCESS KEY" secretKey: "YOUR AWS SECRET KEY"
If Pinot has the correct access credentials to the deep store, then the configured bucket will have the directory created that matches the dataDir.
Getting ideal state and external view for segments from pinot-controller
Enable port-forward for pinot-controller
by running:
kubectl port-forward pinot-controller-0 9000:9000
Ensure that pinot-controller-0
pod is running and fully up by running kubectl get pods
Dump the ideal state and external view for segments for by running:
curl "http://localhost:9000/tables/<tableName>/idealstate" | jq > ideal_state.json 2>&1
curl "http://localhost:9000/tables/<tableName>/externalview" | jq > external_state.json 2>&1
If you do not have jq
or an equivalent tool installed already, follow the installation instructions from here.
Replace <tableName>
with one of the following:
Metrics:
kf_metrics_REALTIME
Events:
kf_events_REALTIME
Logs:
kf_logs_REALTIME
Traces:
kf_traces_REALTIME
For instance, to get the ideal state and external view for logs table, copy-paste the following commands:
curl "http://localhost:9000/tables/kf_logs_REALTIME/idealstate" | jq > ideal_state.json 2>&1
curl "http://localhost:9000/tables/kf_logs_REALTIME/externalview" | jq > external_state.json 2>&1