Kafka Producer CLI - Produce/Send Message to a Topic

In this lesson, you will learn how to produce a message or how to send a message to Kafka’s topic using Kafka Producer CLI.

If your topic does not exist, it is better to create a new topic first.

Table of Contents

Message with a key or without a key

When sending a message to a Kafka topic, you have the option to include a message key or omit it. The key is an optional attribute that can be used to differentiate messages within a topic. Here’s what you need to know:

Sending a message without a key: When you send a message without a key, Kafka will distribute the messages across partitions in a round-robin fashion. This means that each message will be assigned to a different partition in the topic. If you don’t require strict ordering or any specific partitioning scheme, sending messages without keys can be a simple and effective approach. So messages stored without a key are not ordered when you read them from a topic.
Sending a message with a key: Including a key with your message allows you to control how messages are distributed across partitions. Kafka uses the key to determine the partition to which the message should be written. Messages with the same key will always go to the same partition, ensuring that they are processed in order. This can be useful when you need to maintain strict ordering or perform operations based on specific keys.

Sending Message Without a Key

To send a message to the Kafka topic, we use the following command:

./kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

Let’s break it down step by step:

./kafka-console-producer.sh: This part of the command runs the Kafka Console Producer script. The ./ at the beginning indicates that the script is located in the current directory. Usually, this is a bin directory.
--broker-list localhost:9092: This option specifies the list of brokers that the producer should connect to. In this case, it connects to a broker running on localhost (your own machine) and listening to port 9092.When you specify a single broker using the --broker-list parameter, the Kafka producer will only connect to that broker. If that broker goes down, the producer will not be able to send messages to Kafka until the broker is back up and running.On the other hand, when you specify multiple brokers, the producer will connect to all of them. This provides redundancy and fault tolerance in case one or more brokers go down. In addition, it can also help with load balancing by distributing messages across multiple brokers.When specifying multiple brokers, you can separate them using a comma (,) or use multiple --broker-list parameters.
--topic my-topic: This option specifies the name of the topic to which you want to send your message. In this case, it sends the message to a topic named my-topic.

When you run this command, the Kafka Console Producer will start and wait for you to enter input. You can then type your message and press Enter to send it to the specified topic.

Sending Message with a Key

To send a message with a key, we use the following command:

./kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic --property "parse.key=true" --property "key.separator=:"

Let’s break it down step by step:

./kafka-console-producer.sh: This part of the command runs the Kafka Console Producer script. The ./ at the beginning indicates that the script is located in the current directory. Usually, this is a bin directory.
--broker-list localhost:9092: This option specifies the list of brokers I initially want to connect to. In this case, I connect to a broker running on localhost (my local machine). If you have more than one broker running in your cluster, it is better to provide at least two brokers in this list. This is because if one broker is down, Kafka can use another broker in this list to connect to your cluster and send a message.
--topic my-topic: This option specifies the name of the topic to which you want to send your message. In this case, it sends the message to a topic named my-topic.
--property "parse.key=true": This option tells Kafka that you will be sending messages with keys.
--property "key.separator=:": This option specifies the delimiter that separates the key from the value in your message. In this case, it uses a colon (:) as the separator.

When you run this command, the Kafka Console Producer will start and wait for you to enter input. You can then type your message with key-value pairs separated by a colon (:) and press Enter to send it to the specified topic.

Have a look at the image below. The first three messages have the same key “1”. Because these messages have the same key, they will be stored in the same partition. And because they will be stored in the same partition, they will be ordered when I read them.

Messages with different keys will be equally distributed among other topic partitions. This means that when I read those messages, they will not be in strict order.

Kafka Producer CLI with message key

Final words

I hope this tutorial was helpful to you. To learn more about Apache Kafka, check out my other Apache Kafka tutorials for beginners.

Frequently Asked Questions

What happens when I send a message without a key?

When you send a message to a Kafka topic without a key, Kafka will distribute the messages across partitions in a round-robin fashion. This means that each message will be assigned to a different partition in the topic. If you don’t require strict ordering or any specific partitioning scheme, sending messages without keys can be a simple and effective approach.

What are the advantages of specifying multiple brokers?

If you include only one broker in the list and that broker is down, then you can lose the message. However, if you include two brokers in the list and one of them is down, then Kafka can use the healthy broker to still send the message. This makes your system more fault-tolerant and ensures that messages are not lost.