Troubleshooting

Owned by Vinay Krishnamurthy
Last updated: May 09, 2023 by Harold Lim

Networking

Kfuse is unreachable from external host

Symptom

Unable to access Kfuse from the external IP/host or DNS.

curl http://EXTERNAL_IP curl: (28) Failed to connect to XX.XX.XX.XX port 80 after 129551 ms: Connection timed out curl https://EXTERNAL_IP --insecure curl: (28) Failed to connect to XX.XX.XX.XX port 443 after 129551 ms: Connection timed out

Resolution

Ensure that the security group or firewall policy for the Kubernetes cluster, node, and VPC endpoint allows external incoming traffic.

 

Packet is getting dropped by ingress-nginx

Symptom

ingress-nginx logs client intended to send too large body error.

2023/03/06 05:38:22 [error] 43#43: *128072996 client intended to send too large body: 1097442 bytes, client: XXXX, server: _, request: "POST /ingester/v1/fluent_bit HTTP/1.1", host: "XXXX"

Resolution

ingress-nginx can be configured to accept larger request body size. The default is 1m. Upgrade Kfuse with the following section in the custom values file.

ingress-nginx: controller: config: proxy-body-size: <REPLACE THE BODY SIZE HERE, e.g., 8m. Setting to 0 will disable any limit.>

 

Pinot

DeepStore access issues

Symptoms

  • Pinot-related jobs are stuck in crash loop back-off (e.g., kfuse-set-tag-hook, pinot-metrics-table-creation, etc).

  • Pinot-controller logs deep store access-related exception.

    • On AWS S3, the exception has the following format

      Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: MAYE68P6SYZMTTMP, Extended Request ID: L7mSpEzHz9gdxZQ8iNM00jKtoXYhkNrUzYntbbGkpFmUF+tQ8zL+fTpjJRlp2MDLNvhaVYCie/Q=)

Resolution

  • Refer to https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/724664352 for setting the access for Pinot.

    • On GCP, ensure that the secret has correct access to the cloud storage bucket.

    • On AWS S3, if the node does not have permission to the S3 bucket, then ensure that the access key and secret access key is populated

      pinot: deepStore: enabled: true type: "s3" useSecret: true createSecret: true dataDir: "s3://[REPLACE BUCKET HERE]/kfuse/controller/data" s3: region: "YOUR REGION" accessKey: "YOUR AWS ACCESS KEY" secretKey: "YOUR AWS SECRET KEY"

  • If Pinot has the correct access credentials to the deep store, then the configured bucket will have the directory created that matches the dataDir.



Getting ideal state and external view for segments from pinot-controller

Enable port-forward for pinot-controller by running:

kubectl port-forward pinot-controller-0 9000:9000

Ensure that pinot-controller-0 pod is running and fully up by running kubectl get pods

Dump the ideal state and external view for segments for by running:

curl "http://localhost:9000/tables/<tableName>/idealstate" | jq > ideal_state.json 2>&1 curl "http://localhost:9000/tables/<tableName>/externalview" | jq > external_state.json 2>&1

If you do not have jq or an equivalent tool installed already, follow the installation instructions from here.

Replace <tableName> with one of the following:

  • Metrics: kf_metrics_REALTIME

  • Events: kf_events_REALTIME

  • Logs: kf_logs_REALTIME

  • Traces: kf_traces_REALTIME

For instance, to get the ideal state and external view for logs table, copy-paste the following commands:

curl "http://localhost:9000/tables/kf_logs_REALTIME/idealstate" | jq > ideal_state.json 2>&1 curl "http://localhost:9000/tables/kf_logs_REALTIME/externalview" | jq > external_state.json 2>&1