In this lesson, you will learn how to produce a message or how to send a message to Kafka’s topic using Kafka Producer CLI.
If your topic does not exist, it is better to create a new topic first.
Message with a key or without a key
When sending a message to a Kafka topic, you have the option to include a message key or omit it. The key is an optional attribute that can be used to differentiate messages within a topic. Here’s what you need to know:
- Sending a message without a key: When you send a message without a key, Kafka will distribute the messages across partitions in a round-robin fashion. This means that each message will be assigned to a different partition in the topic. If you don’t require strict ordering or any specific partitioning scheme, sending messages without keys can be a simple and effective approach. So messages stored without a key are not ordered when you read them from a topic.
- Sending a message with a key: Including a key with your message allows you to control how messages are distributed across partitions. Kafka uses the key to determine the partition to which the message should be written. Messages with the same key will always go to the same partition, ensuring that they are processed in order. This can be useful when you need to maintain strict ordering or perform operations based on specific keys.
Sending Message Without a Key
To send a message to the Kafka topic, we use the following command:
./kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic
Let’s break it down step by step:
./kafka-console-producer.sh
: This part of the command runs the Kafka Console Producer script. The./
at the beginning indicates that the script is located in the current directory. Usually, this is a bin directory.--broker-list localhost:9092
: This option specifies the list of brokers that the producer should connect to. In this case, it connects to a broker running onlocalhost
(your own machine) and listening to port9092
.When you specify a single broker using the--broker-list
parameter, the Kafka producer will only connect to that broker. If that broker goes down, the producer will not be able to send messages to Kafka until the broker is back up and running.On the other hand, when you specify multiple brokers, the producer will connect to all of them. This provides redundancy and fault tolerance in case one or more brokers go down. In addition, it can also help with load balancing by distributing messages across multiple brokers.When specifying multiple brokers, you can separate them using a comma (,
) or use multiple--broker-list
parameters.--topic my-topic
: This option specifies the name of the topic to which you want to send your message. In this case, it sends the message to a topic namedmy-topic
.
When you run this command, the Kafka Console Producer will start and wait for you to enter input. You can then type your message and press Enter to send it to the specified topic.
Sending Message with a Key
To send a message with a key, we use the following command:
./kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic --property "parse.key=true" --property "key.separator=:"
Let’s break it down step by step:
./kafka-console-producer.sh
: This part of the command runs the Kafka Console Producer script. The./
at the beginning indicates that the script is located in the current directory. Usually, this is a bin directory.--broker-list localhost:9092
: This option specifies the list of brokers I initially want to connect to. In this case, I connect to a broker running onlocalhost
(my local machine). If you have more than one broker running in your cluster, it is better to provide at least two brokers in this list. This is because if one broker is down, Kafka can use another broker in this list to connect to your cluster and send a message.--topic my-topic
: This option specifies the name of the topic to which you want to send your message. In this case, it sends the message to a topic namedmy-topic
.--property "parse.key=true"
: This option tells Kafka that you will be sending messages with keys.--property "key.separator=:"
: This option specifies the delimiter that separates the key from the value in your message. In this case, it uses a colon (:
) as the separator.
When you run this command, the Kafka Console Producer will start and wait for you to enter input. You can then type your message with key-value pairs separated by a colon (:
) and press Enter to send it to the specified topic.
Have a look at the image below. The first three messages have the same key “1”. Because these messages have the same key, they will be stored in the same partition. And because they will be stored in the same partition, they will be ordered when I read them.
Messages with different keys will be equally distributed among other topic partitions. This means that when I read those messages, they will not be in strict order.
Final words
I hope this tutorial was helpful to you. To learn more about Apache Kafka, check out my other Apache Kafka tutorials for beginners.
Frequently Asked Questions
What happens when I send a message without a key?
When you send a message to a Kafka topic without a key, Kafka will distribute the messages across partitions in a round-robin fashion. This means that each message will be assigned to a different partition in the topic. If you don’t require strict ordering or any specific partitioning scheme, sending messages without keys can be a simple and effective approach.
What are the advantages of specifying multiple brokers?
If you include only one broker in the list and that broker is down, then you can lose the message. However, if you include two brokers in the list and one of them is down, then Kafka can use the healthy broker to still send the message. This makes your system more fault-tolerant and ensures that messages are not lost.