In this tutorial, you will learn how to start Apache Kafka Server with KRaft, a new consensus protocol that replaces ZooKeeper for metadata management. KRaft simplifies Kafka’s architecture and improves its performance and stability. You can read more about KRaft here and here.
Step 1: Generate a Cluster UUID
Before you can start Apache Kafka server on your computer, you need to generate a UUID for your Kafka cluster. A cluster UUID is a unique identifier for your Kafka cluster. You can generate one by running the following command in your terminal:
This will store the cluster UUID in a variable called
KAFKA_CLUSTER_ID. You can print it by running:
Note: All the commands in this tutorial assume that you are in the root directory of the Kafka folder that you downloaded and extracted. If you are in a different location, you may need to adjust the paths accordingly.
Step 2: Format Log Directories
Now that you have generated a unique identified for your Kafka cluster, you can use it to format the log directories where Kafka will store its data. You can do this by running the following command:
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties
This will use the cluster UUID and the configuration file
config/kraft/server.properties to format the log directories. You can modify the configuration file to change the default settings, such as the log directory path, the number of controllers, and the ports.
What are log directories?
Log directories are the folders where Kafka stores its data as ordered sequences of messages called logs. Each log is divided into smaller files called segments, which contain the actual messages and some index files to help locate them. Log directories are organized by topics and partitions, which are logical groups of logs that belong to a specific use case and provide parallelism and fault tolerance.
Why do we need to format log directories?
We need to format log directories before we can start Kafka in KRaft mode. Formatting log directories means assigning a unique identifier to the cluster and initializing the metadata for the partitions. This prevents accidental configuration changes and ensures compatibility with KRaft mode.
What is server.properties file?
The server.properties is a configuration file that specifies various settings for the Kafka server, such as the broker id, the listeners, the log directories, the number of controllers, the ports, and the log retention policies. You can modify this file to change the default values according to your needs and preferences. You can also override some of the properties by passing them as command-line arguments when starting the Kafka server.
Step 3: Start the Kafka Server
Now you are ready to start Apache Kafka server in KRaft mode. You can do this by running the following command:
This will start the Kafka server as both a broker and a controller in a single process. This is suitable for local development and testing, but not for production environments. For production, you should run separate processes for brokers and controllers, and use multiple nodes for fault tolerance.
Note: By default, when starting a single Kafka server on your computer, it will listen on
localhost:9092 for incoming connections from clients. You can change this by modifying the
listeners property in the
Step 4: Test the Kafka Server
To test if the Kafka server is working properly, you can use the Kafka command-line tools to create a topic, produce some messages, and consume them. For example, you can run the following commands in separate terminals:
# Create a topic named product-created-events-topic with one partition and one replica bin/kafka-topics.sh --create --topic product-created-events-topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092 # Produce some messages to the topic bin/kafka-console-producer.sh --topic product-created-events-topic --bootstrap-server localhost:9092 # Consume the messages from the topic bin/kafka-console-consumer.sh --topic product-created-events-topic --from-beginning --bootstrap-server localhost:9092
You should see the messages you typed in the producer terminal appear in the consumer terminal.
I hope this tutorial is helpful to you. To learn more about Apache Kafka, please check my other Apache Kafka tutorials for beginners. You can also find more resources and documentation on the official Kafka website.