OnCallReadyStart learning

Runtimes and data Lesson 28.9 · Kafka Operations · 14 min read

Producing and consuming from the command line

In plain words

The console producer is a typewriter connected to a topic: every line you type becomes a record. The console consumer is a reading lamp that shows the records as they arrive. A key is like a name on an envelope: every envelope with the same name goes into the same drawer, so letters to one person always stay in order.

Talking to Kafka from the shell

The shop's services use the Java client library. You will rarely write a producer, but during an incident you need to answer "is anything being written?", "what does that record look like?" and "does a write work at all?" - from a terminal. The distribution ships two tools for that, built on the same client library: kafka-console-producer.sh and kafka-console-consumer.sh.

Producing

The console producer reads standard input: one line = one record. Interactively it shows a > prompt; Ctrl+D ends it. In a pipe, it sends every line and exits:

$ printf 'first line\nsecond line\n' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic notifications

No output means success. Without a key the producer uses the sticky partitioner: a batch of records goes to one partition, the next batch to another, which spreads load without tiny batches.

With keys, --reader-property parse.key=true splits each line at the key separator, a TAB by default (key.separator changes it):

$ printf 'c-1001:{"plan":"gold"}\nc-1002:{"plan":"free"}\nc-1001:{"plan":"platinum"}\n' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic notifications --reader-property parse.key=true --reader-property key.separator=:

A keyed record always goes to the same partition: murmur2(key bytes), made positive, modulo the number of partitions. That is how per-key ordering works (every update of customer c-1001 lands in one partition, in order) - and why adding partitions breaks it. A line without the separator stops the producer:

$ printf 'no separator here\n' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic notifications --reader-property parse.key=true --reader-property key.separator=:
org.apache.kafka.common.KafkaException: No key separator found on line number 1: 'no separator here'
	at org.apache.kafka.tools.LineMessageReader.parse(LineMessageReader.java:177)
	at org.apache.kafka.tools.LineMessageReader$1.hasNext(LineMessageReader.java:132)
	at org.apache.kafka.tools.ConsoleProducer.loopReader(ConsoleProducer.java:99)
	at org.apache.kafka.tools.ConsoleProducer.start(ConsoleProducer.java:71)
	at org.apache.kafka.tools.ConsoleProducer.main(ConsoleProducer.java:60)

Two more options you will use: --request-required-acks (the acks setting, -1 = all by default) and --command-property key=value for any producer setting (--producer-property still works but is deprecated since 4.2). A record to a topic that does not exist, with automatic creation off, waits max.block.ms (60 s) and fails with Topic ... not present in metadata after 60000 ms..

Consuming

The console consumer prints the values, one per line, until you press Ctrl+C - then it says how many it processed (on stderr). Two options make it stop by itself: --max-messages N and --timeout-ms (exit after that long without a new record).

$ kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic notifications --max-messages 3 --timeout-ms 5000
The consumer rebalance protocol (KIP-848) is production-ready! Set group.protocol=consumer to try it out. See https://kafka.apache.org/documentation/#consumer_rebalance_protocol
[2026-09-22 20:00:17,800] ERROR Error processing message, terminating consumer process:  (org.apache.kafka.tools.consumer.ConsoleConsumer)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 0 messages

Nothing came (nothing is writing to notifications right now), and without --timeout-ms the command would have waited for ever: a new consumer starts at the end of the log (auto.offset.reset=latest). The timeout ends it with an ERROR line and a TimeoutException - that is how the tool reports "nothing arrived", not a failure of the cluster. That default surprises everyone once. --from-beginning starts at the oldest record still in the log - only if the group has no committed offset yet:

$ kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic notifications --from-beginning --max-messages 3
The consumer rebalance protocol (KIP-848) is production-ready! Set group.protocol=consumer to try it out. See https://kafka.apache.org/documentation/#consumer_rebalance_protocol
{"to":"c-1000","template":"order_confirmed","order_id":"o-100000","ts":"2026-09-21T08:00:03.000Z"}
{"to":"c-1049","template":"invoice_ready","order_id":"o-100007","ts":"2026-09-21T08:00:03.250Z"}
{"to":"c-1098","template":"shipped","order_id":"o-100014","ts":"2026-09-21T08:00:03.500Z"}
Processed a total of 3 messages

The first line on stderr is the tool advertising the new consumer group protocol (KIP-848, next lesson); the console consumer prints it whenever you use the default "classic" protocol.

What a record carries besides the value comes out with --formatter-property (--property in older guides, now deprecated):

$ kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic notifications --from-beginning --max-messages 2 --formatter-property print.timestamp=true --formatter-property print.partition=true --formatter-property print.offset=true --formatter-property print.key=true
The consumer rebalance protocol (KIP-848) is production-ready! Set group.protocol=consumer to try it out. See https://kafka.apache.org/documentation/#consumer_rebalance_protocol
CreateTime:1789977603000	Partition:0	Offset:0	c-1000	{"to":"c-1000","template":"order_confirmed","order_id":"o-100000","ts":"2026-09-21T08:00:03.000Z"}
CreateTime:1789977603250	Partition:0	Offset:1	c-1049	{"to":"c-1049","template":"invoice_ready","order_id":"o-100007","ts":"2026-09-21T08:00:03.250Z"}
Processed a total of 2 messages

The fields come in a fixed order - timestamp, partition, offset, key, value - separated by a TAB (key.separator changes it). CreateTime: is the producer's timestamp; a topic with message.timestamp.type=LogAppendTime shows the broker's arrival time instead.

To read one partition from one offset, give both:

$ kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic notifications --partition 0 --offset earliest --max-messages 2 --formatter-property print.offset=true
Offset:0	{"to":"c-1000","template":"order_confirmed","order_id":"o-100000","ts":"2026-09-21T08:00:03.000Z"}
Offset:1	{"to":"c-1049","template":"invoice_ready","order_id":"o-100007","ts":"2026-09-21T08:00:03.250Z"}
Processed a total of 2 messages

--partition cannot be combined with --group: reading by hand from a fixed position is the opposite of group consumption.

The console consumer is a group member too

Without --group, the consumer joins a throw-away group called console-consumer-<random number> with automatic commits turned off. With --group billing-debug it joins (or creates) that group and commits what it read. Never point a debugging consumer at a production group's name: it takes partitions away from the real consumers (a rebalance), and its commits move the group's position.

Where are the ends?

kafka-get-offsets.sh asks each partition's leader for an offset:

$ kafka-get-offsets.sh --bootstrap-server kafka-1:9092 --topic notifications
notifications:0:518402
notifications:1:518402
notifications:2:518401
$ kafka-get-offsets.sh --bootstrap-server kafka-1:9092 --topic notifications --time earliest
notifications:0:0
notifications:1:0
notifications:2:0

The default (--time latest, -1) is the end of the log (the high watermark: the next offset to be written); earliest (-2) is the first offset still on disk - retention moves it forward. The difference is how many records each partition holds right now. A timestamp in milliseconds instead returns the first offset at or after that time, which is exactly what an offset reset to a date uses (lesson 7).

In an interview: "A new consumer starts at the end unless the group has a committed offset or you ask for earliest. A keyed record always lands in the same partition, which is what gives per-key ordering."

Why it helps

During an incident you need to answer "can I write at all?", "what does a record look like?" and "where does this key go?" without writing code. The console clients answer those in seconds. Knowing their defaults - start at the end, wait forever, print values only - prevents wrong conclusions such as "the topic is empty" when it is not.

Commands in this lesson

printf kafka-console-consumer.sh kafka-get-offsets.sh

FAQ

Why does kafka-console-consumer print nothing?

By default it starts at the end of the log (auto.offset.reset=latest) and waits for new records, so on a quiet topic it shows nothing. Add --from-beginning to start at the oldest record, and --timeout-ms or --max-messages so the command ends by itself.

How do I send a record with a key?

With --reader-property parse.key=true, each input line is split at the key separator, a TAB by default; --reader-property key.separator=: changes it. A line without the separator stops the producer with an error naming the line number.

Which partition does a key go to?

murmur2 of the key bytes, made positive, modulo the number of partitions. Every client using the default partitioner computes the same answer, so all records with one key go to one partition and stay in order there.

What is --formatter-property for?

It turns on extra fields in the consumer's output: print.timestamp, print.partition, print.offset, print.key and others. They come in a fixed order, separated by a TAB. The older --property option still works but is deprecated, so new scripts should use the new name.

Why should I not use a production group name with the console consumer?

With --group the console consumer joins that group: it takes partitions away from the real consumers in a rebalance and commits its own offsets, which moves the group's position. For debugging, leave --group out so it uses a throw-away group.

In an interview Mid

How do you check from the command line whether a topic receives data and what it looks like?

kafka-get-offsets.sh shows the end offset of each partition - run it twice to see whether it grows. kafka-console-consumer.sh with --from-beginning, --max-messages and print.key, print.partition and print.offset shows real records. kafka-console-producer.sh with acks=all is a quick write test. A keyed record always lands in the same partition, which is what gives per-key ordering.

Also asked: Why does a new consumer start at the end? · How does a key choose its partition? · What happens to key ordering when partitions are added?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.