Kfuse Custom YAML changes

Update the global.kafkaTopics section with the updated replicationFactor. Note that this does not actually take into effect on an existing deployed cluster because Kafka does not support changing replicationFactor out of the box with a single command. This is only needed for tracking purposes and if using the yaml to re-deploy a fresh installation.

Resize Kafka Persistent Disk (If Needed)

By default, Kfuse uses 10GB retention size per partition (refer to kafka.logRetentionBytes settings).

Ensure that the

Ensure that the persistent disk size has enough capacity for the (number of partitions * replicationFactor across all topics defined in the global.kafkaTopics * logRetentionBytes ) / (number of Kafka Brokers).
If the currently configured persistent disk needs to be increased, refer to Troubleshooting | Increasing the existing PVC size

Steps to Increase ReplicationFactor

Login to Kafka Pod

Unset JMX_PORT

Get the list of configured topics (or get the list from global.kafkaTopics)

Create topics.json file (ensure that this file is saved in /bitnami/kafka)

Get the current assignment (Note that the --broker-list should match the number of current configured brokers) The example command below is for setup with 3 Kafka brokers

The above command will print out the Current partition replica assignment and Proposed partition reassignment configuration. Ignore the proposed output. Only the Current partition replica assignment is needed.

Copy current partition replica assignment to a file.

Example output

Create a copy of the partition replica assignment file and modify the replicas and log_dirs field.

For each replicas field, add N number of brokers (depending on the replicationFactor desired. Note that the log_dirs field needs to match. For each new broker added in the replicas field, add a "any" item in the log_dirs field. Ensure that the replicas are balanced across all brokers.

For example (focusing on 1 parition of kf_metrics_topic)

Save the new assignment files in /bitnami/kafka

Run the reassignment

Verify reassignment