Kfuse Custom YAML changes
Update the global.kafkaTopics section with the updated replicationFactor. Note that this does not actually take into effect on an existing deployed cluster because Kafka does not support changing replicationFactor out of the box with a single command. This is only needed for tracking purposes and if using the yaml to re-deploy a fresh installation.
Resize Kafka Persistent Disk (If Needed)
By default, Kfuse uses 10GB retention size per partition (refer to kafka.logRetentionBytes settings).
Ensure that the
Ensure that the persistent disk size has enough capacity for the (number of partitions * replicationFactor across all topics defined in the global.kafkaTopics * logRetentionBytes ) / (number of Kafka Brokers).
If the currently configured persistent disk needs to be increased, refer to https://kloudfuse.atlassian.net/wiki/spaces/EX/pages/786038817/Troubleshooting#Increasing-the-existing-PVC-size
Steps to Increase ReplicationFactor
Login to Kafka Pod
kubectl exec -ti -n metrics kafka-0 -- bash
Unset JMX_PORT
unset JMX_PORT
Get the list of configured topics (or get the list from global.kafkaTopics)
/opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server :9092 --list
Create topics.json file (ensure that this file is saved in /bitnami/kafka)
cat > /bitnami/kafka/topics.json
{
"version": 1,
"topics": [
{ "topic": "kf_events_topic" },
{ "topic": "kf_logs_metric_topic" },
{ "topic": "kf_logs_topic" },
{ "topic": "kf_metrics_topic" },
{ "topic": "kf_traces_errors_topic" },
{ "topic": "kf_traces_metric_topic" },
{ "topic": "kf_traces_topic" },
{ "topic": "logs_ingest_topic" }
]
}
Get the current assignment (Note that the --broker-list should match the number of current configured brokers) The example command below is for setup with 3 Kafka brokers
/opt/bitnami/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server :9092 --generate --topics-to-move-json-file /bitnami/kafka/topics.json --broker-list 0,1,2
The above command will print out the Current partition replica assignment and Proposed partition reassignment configuration. Ignore the proposed output. Only the Current partition replica assignment is needed.
Copy current partition replica assignment to a file.
Example output
{"version":1,"partitions":[{"topic":"kf_events_topic","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_logs_metric_topic","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_logs_topic","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":3,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":4,"replicas":[1],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":5,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":6,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":7,"replicas":[1],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":8,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":9,"replicas":[2],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":10,"replicas":[1],"log_dirs":["any"]},{"topic":"kf_metrics_topic","partition":11,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_traces_errors_topic","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_traces_metric_topic","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"kf_traces_topic","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"logs_ingest_topic","partition":0,"replicas":[0],"log_dirs":["any"]}]}
Create a copy of the partition replica assignment file and modify the replicas and log_dirs field.
For each replicas field, add N number of brokers (depending on the replicationFactor desired. Note that the log_dirs field needs to match. For each new broker added in the replicas field, add a "any" item in the log_dirs field. Ensure that the replicas are balanced across all brokers.
For example (focusing on 1 parition of kf_metrics_topic)
...
{"topic":"kf_metrics_topic","partition":3,"replicas":[2,0],"log_dirs":["any", "any"]}
... snipped ...
...
Save the new assignment files in /bitnami/kafka
cat > /bitnami/kafka/topics.assignment.json <PASTE THE new assignments here>
Run the reassignment
/opt/bitnami/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server :9092 --execute --reassignment-json-file /bitnami/kafka/topics.assignment.json
Verify reassignment
/opt/bitnami/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server :9092 --verify --reassignment-json-file /bitnami/kafka/topics.assignment.json