In this tutorial, you will learn how to create a new Topic using Kafka Command Line Interface(CLI).
1. Open a new Terminal Window
To create a new Kafka topic, using CLI you will need to use a terminal window or command-line prompt.
2. Change Directory to Kafka “bin” Folder
Once you have a terminal window opened, change directory to your Kafka folder.
Once you are inside a Kafka folder, list files there. You should see a bin folder. Kafka scripts are located in the bin folder. So, Let’s change directory to a bin folder.
If you list files in the bin folder. You will see a list of different scripts that you can use to work with Kafka.
Notice that all these scripts have extension .sh. This means that these files will work on Linux and Mac operating systems.
2.1 On Windows
If you are on Windows, then you need to change directory to a windows directory. It is also inside the bin folder.
If you list files here, then you should see the same kafka scripts but with .bat file extension. These scripts should work on windows.
Creating a new Kafka Topic
To create a new Kafka topic I will use the following command.
./kafka-topics.sh --create --topic topic1 --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
The above command has several parameters that specify the properties of the topic, such as:
- –create: This tells Kafka that you want to create a new topic, not modify or delete an existing one.
- –topic topic1: This tells Kafka the name of the topic you want to create. You can choose any name you like, as long as it is unique and does not contain any special characters. In this case, I have chosen topic1 as the name.
- –partitions 3: This tells Kafka how many partitions you want to create for the topic. Partitions are smaller units of data that can be distributed across different servers or brokers. Having more partitions can improve the performance and scalability of your topic, but also increase the complexity and overhead. In this case, I have chosen to create 3 partitions for the topic.
- –replication-factor 1: This tells Kafka how many replicas you want to create for each partition. Replicas are copies of the data in the partition that are stored on other brokers for fault tolerance. Having more replicas can improve the reliability and availability of your topic, but also increase the storage and network usage. In this case, I have chosen to create 1 replica for each partition.
IMPORTANT: The number you specify in the –replication-factor parameter, cannot be greater than the number of brokers you have in your Kafka cluster. So if you started only a one, single kafka broker, then you cannot have replication factor greater than 1. But if you start three brokers in a cluster, then you can specify replication factor 3. And each partition will have a copy in each broker. - –bootstrap-server localhost:9092: This parameter specifies a list of addresses of Kafka brokers in a Kafka cluster. Here I can provide multiple servers separated with comma. And even though I have multiple servers running in the cluster, I can provide the address of one, initial server only. If this initial server is healthy, Kafka client will still be able to discover other servers and access all the topics and partitions.But, if this initial broker is not healthy, if it goes down or becomes unreachable, you will lose the connection and won’t be able to reconnect to the cluster until this broker is back online. This can cause data loss, latency, or unavailability issues for your application. Therefore, it is recommended to provide the addresses of at least two brokers with the –bootstrap-server parameter, so that you can have a backup broker in case one broker is not available.
Once you execute the above command, a new Kafka topic should be created.
