In this tutorial, I want to share with you how to add or update the configuration of an existing Kafka topic.
You may need to update the configuration of an existing Kafka topic when you want to change the behaviour of the topic or when you want to optimize the performance of your Kafka cluster.
For example, you may want to increase the minimum number of in-sync replicas required for a producer to receive an acknowledgment of a write operation, or you may want to increase the maximum size of a message that can be written to the Kafka topic.
Before Updating Kafka Topic Configuration
Before updating the configuration of an Apache Kafka topic, it is generally recommended to follow these steps:
- Consult your team members about the changes you are going to make,
- Stop producers and consumers that are using the target topic(s). This ensures that no data is being written to or read from the topic during the configuration update process.
Once the topic configuration is updated, you can start Kafka producers and consumers to test if your application works as expected.
It’s important to note that these steps may vary depending on your specific Kafka setup and deployment environment.
Updating Kafka Topic Configuration
Let’s assume that we need to update the Kafka topic configuration to change the minimum number of in-sync replicas. To do that, we will use the kafka-configs.sh script, which you will find in your Apache Kafka’s “bin” folder.
windows/kafka-configs.bat --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name topic-insync --add-config min.insync.replicas=2
On MacOS or Linux
./bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name topic-insync --add-config min.insync.replicas=2
./bin/kafka-configs.sh: This is the path to the
kafka-configs.shscript that you need to run in order to update the configuration of a Kafka topic.
--bootstrap-server localhost:9092: This specifies the Kafka broker(s) that you want to connect to. In this case, I am connecting to a broker running on
--alter: This tells the script that you want to alter the configuration of an existing Kafka topic.
--entity-type topics: This specifies that you want to alter the configuration of a Kafka topic.
--entity-name topic-insync: This specifies the name of the Kafka topic that you want to update. In my case, I am updating the topic with the name “topic-insync”,
--add-config min.insync.replicas=2: This specifies the configuration property that you want to update and its new value. In this case, I am updating the
min.insync.replicasproperty to have a value of
What topic configuration can be updated?
kafka-configs.sh script cannot be used to update just any information about the topic. For example, it cannot update the topic name.
Below is a list of some of the configuration properties that you can update using the
- min.insync.replicas: The minimum number of in-sync replicas required for a producer to receive an acknowledgment of a write operation,
- retention.ms: The time duration for which a message will be retained in the Kafka topic,
- max.message.bytes: The maximum size of a message that can be written to the Kafka topic,
- cleanup.policy: The policy used to determine when messages should be deleted from the Kafka topic,
- segment.bytes: The size of each log segment file,
- Other properties.
Please note that this is not a complete list, and there are many other configuration properties that you can update using the
kafka-configs.sh script. For a complete list of topic configurations, I have included a link to Confluent documentation called “Other properties”.
How do we update the Apache Kafka topic name?
At the moment of writing this tutorial, renaming an existing Apache Kafka topic is not supported. You cannot use
kafka-configs.sh to change the name of an existing Kafka topic. But there is a way around it.
Basically, you need to create a new topic and copy all messages from the old topic to a new one.
Follow these steps:
- Create a new topic with a new name,
- Use a Kafka consumer to read all messages from the old topic,
- Use a Kafka producer to write all messages to the new topic,
- Once all messages have been copied to a new topic, you can delete the old topic.
But before doing it, consider the amount of messages that need to be transferred from an old topic to a new one. If there are too many messages, the transfer can consume additional resources and maybe keeping an old topic name is a better option.
I hope this tutorial was helpful to you. To learn more about Kafka, please check my other Apache Kafka tutorials.