Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Kfuse Custom YAML changes

Update the global.kafkaTopics section with the updated replicationFactor. Note that this does not actually take into effect on an existing deployed cluster because Kafka does not support changing replicationFactor out of the box with a single command. This is only needed for tracking purposes and if using the yaml to re-deploy a fresh installation.

Resize Kafka Persistent Disk (If Needed)

By default, Kfuse uses 10GB retention size per partition (refer to kafka.logRetentionBytes settings).

Ensure that the

Ensure that the persistent disk size has enough capacity for the (number of partitions * replicationFactor across all topics defined in the global.kafkaTopics * logRetentionBytes ) / (number of Kafka Brokers).
If the currently configured persistent disk needs to be increased, refer to https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/786038817/Troubleshooting#Increasing-the-existing-PVC-size

Steps to Increase ReplicationFactor

Login to Kafka Pod

kubectl exec -ti -n metrics kafka-0 -- bash

Unset JMX_PORT

 unset JMX_PORT

Get the list of configured topics (or get the list from global.kafkaTopics)

/opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server :9092 --list

Create topics.json file (ensure that this file is saved in /bitnami/kafka)

cat > /bitnami/kafka/topics.json
{
  "version": 1,
  "topics": [
    { "topic": "kf_events_topic" },
    { "topic": "kf_logs_metric_topic" },
    { "topic": "kf_logs_topic" },
    { "topic": "kf_metrics_topic" },
    { "topic": "kf_traces_errors_topic" },
    { "topic": "kf_traces_metric_topic" },
    { "topic": "kf_traces_topic" },
    { "topic": "logs_ingest_topic" }
  ]
}

Get the current assignment (Note that the --broker-list should match the number of current configured brokers) The example command below is for setup with 3 Kafka brokers

/opt/bitnami/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server :9092 --generate --topics-to-move-json-file /bitnami/kafka/topics.json  --broker-list 0,1,2

The above command will print out the Current partition replica assignment and Proposed partition reassignment configuration. Ignore the proposed output. Only the Current partition replica assignment is needed.

Copy current partition replica assignment to a file.

Example output

{"version":1,"partitions":[{"topic":"kf_events_topic","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_logs_metric_topic","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_logs_topic","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":3,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":4,"replicas":[1],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":5,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":6,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":7,"replicas":[1],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":8,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":9,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":10,"replicas":[1],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":11,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_traces_errors_topic","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_traces_metric_topic","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_traces_topic","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"logs_ingest_topic","partition":0,"replicas":[0],"log_dirs":["any"]}]}

Create a copy of the partition replica assignment file and modify the replicas and log_dirs field.

For each replicas field, add N number of brokers (depending on the replicationFactor desired. Note that the log_dirs field needs to match. For each new broker added in the replicas field, add a "any" item in the log_dirs field.

For example (focusing on 1 parition of kf_metrics_topic)

...
{"topic":"kf_metrics_topic","partition":3,"replicas":[2,0],"log_dirs":["any", "any"]}
... snipped ...
...

Save the new assignment files in /bitnami/kafka

cat > /bitnami/kafka/topics.assignment.json
<PASTE THE new assignments here>

Run the reassignment

/opt/bitnami/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server :9092 --execute --reassignment-json-file /bitnami/kafka/topics.assignment.json

Verify reassignment

/opt/bitnami/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server :9092 --verify --reassignment-json-file /bitnami/kafka/topics.assignment.json
  • No labels