In this lesson, you will learn how to consume messages or how to read messages from Apache Kafka topic using Kafka Consumer Command Line Interface(CLI).
To learn how to produce or send messages to Kafka topic, please read the following tutorial: Kafka Producer CLI – Produce/Send message to Kafka topic.
Consuming Messages From the Beginning
To consume or read all messages from the Apache Kafka topic using Kafka CLI, you will need to first open a terminal window and change the directory to the Kafka folder. Once you are in the Kafka folder, type the following command:
bin/windows/kafka-console-consumer.bat --topic my-topic --from-beginning --bootstrap-server localhost:9092
On Mac or Linux
bin/kafka-console-consumer.sh --topic my-topic --from-beginning --bootstrap-server localhost:9092
Let’s break down the command:
bin/kafka-console-consumer.sh: This part of the command specifies the Kafka command-line consumer script that you want to use.
--topic my-topic: This part specifies the Kafka topic from which you want to consume messages. Replace “my-topic” with the actual name of the topic you’re interested in.
--from-beginning: This flag instructs the consumer to start consuming messages from the beginning of the topic’s log. In other words, it will read all the messages in the topic, starting with the earliest available message, and continue to consume new messages as they are produced to the topic.
--bootstrap-server localhost:9092with this flag, you are providing the consumer script with the initial Kafka bootstrap server. If you have multiple servers running in your Kafka cluster, then, with this parameter, you will provide the address of one of those servers as an initial server to connect to. Once the consumer script connects to one server, it will be able to discover other servers as well. If needed, you can provide two servers here separating them with commas. This way if one server is not available, Kafka consumer will use a second server to connect to the cluster.
Important: Every time you run the consumer script with
--from-beginning parameter, it will read messages from the beginning. Even if you run it multiple times. It will always consume messages from Kafka topic from the beginning. In the following section, you will learn how to read recent messages only.
To learn about other command-line arguments that Kafka Console Consumer can accept, check this documentation page.
Reading New Messages Only
To read new Kafka messages from the Kafka topic, you simply do not use the
--from-beginning parameter. So your new command will look like this:
bin/kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092
By simply omitting
--from-beginning, you’re telling Kafka, “I’m interested in this topic, but I only want to read new messages, not the old ones that were sent before I started reading.”
How to Display Message Key?
Messages in Kafka topic are stored as a key:value pairs. To display both a message and a message key, you can use the following two parameters:
--property print.key=truethis tells the consumer to print the message keys,
--property print.value=truethis tells the consumer to print the message.
So your consumer console command will look like this:
bin/kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092 --property print.key=true --property print.value=true
How to Stop Kafka Consumer CLI Script?
To stop the consumer CLI script, simply press
Ctrl + C in your terminal. This will halt the consumer and allow you to exit gracefully.
How to Create Kafka Consumer Group in CLI?
To make your Kafka Console Consumer join a consumer group you can use the
--group option is used to specify the name of the consumer group that the consumer belongs to. When you run a consumer, it will automatically join a consumer group with a default name. However, you can override this default name by specifying your own group name using the
./kafka-console-consumer.sh --topic product-created-topic --bootstrap-server localhost:9092 --from-beginning --property print.key=true --property print.value=true --group my-created-consumer-group
The consumer will join a consumer group named
my-created-consumer-group. This is useful when you want to have multiple consumers that belong to different groups.
I hope this tutorial was helpful to you. To learn more about Apache Kafka, check out my other Apache Kafka tutorials for beginners.