Talking to Kafka from the shell
The shop's services use the Java client library. You will rarely write a producer, but during an incident you need to answer "is anything being written?", "what does that record look like?" and "does a write work at all?" - from a terminal. The distribution ships two tools for that, built on the same client library: kafka-console-producer.sh and kafka-console-consumer.sh.
Producing
The console producer reads standard input: one line = one record. Interactively it shows a > prompt; Ctrl+D ends it. In a pipe, it sends every line and exits:
$ printf 'first line\nsecond line\n' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic notifications
No output means success. Without a key the producer uses the sticky partitioner: a batch of records goes to one partition, the next batch to another, which spreads load without tiny batches.
With keys, --reader-property parse.key=true splits each line at the key separator, a TAB by default (key.separator changes it):
$ printf 'c-1001:{"plan":"gold"}\nc-1002:{"plan":"free"}\nc-1001:{"plan":"platinum"}\n' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic notifications --reader-property parse.key=true --reader-property key.separator=:
A keyed record always goes to the same partition: murmur2(key bytes), made positive, modulo the number of partitions. That is how per-key ordering works (every update of customer c-1001 lands in one partition, in order) - and why adding partitions breaks it. A line without the separator stops the producer:
$ printf 'no separator here\n' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic notifications --reader-property parse.key=true --reader-property key.separator=:
org.apache.kafka.common.KafkaException: No key separator found on line number 1: 'no separator here'
at org.apache.kafka.tools.LineMessageReader.parse(LineMessageReader.java:177)
at org.apache.kafka.tools.LineMessageReader$1.hasNext(LineMessageReader.java:132)
at org.apache.kafka.tools.ConsoleProducer.loopReader(ConsoleProducer.java:99)
at org.apache.kafka.tools.ConsoleProducer.start(ConsoleProducer.java:71)
at org.apache.kafka.tools.ConsoleProducer.main(ConsoleProducer.java:60)
Two more options you will use: --request-required-acks (the acks setting, -1 = all by default) and --command-property key=value for any producer setting (--producer-property still works but is deprecated since 4.2). A record to a topic that does not exist, with automatic creation off, waits max.block.ms (60 s) and fails with Topic ... not present in metadata after 60000 ms..
Consuming
The console consumer prints the values, one per line, until you press Ctrl+C - then it says how many it processed (on stderr). Two options make it stop by itself: --max-messages N and --timeout-ms (exit after that long without a new record).
$ kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic notifications --max-messages 3 --timeout-ms 5000
The consumer rebalance protocol (KIP-848) is production-ready! Set group.protocol=consumer to try it out. See https://kafka.apache.org/documentation/#consumer_rebalance_protocol
[2026-09-22 20:00:17,800] ERROR Error processing message, terminating consumer process: (org.apache.kafka.tools.consumer.ConsoleConsumer)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 0 messages
Nothing came (nothing is writing to notifications right now), and without --timeout-ms the command would have waited for ever: a new consumer starts at the end of the log (auto.offset.reset=latest). The timeout ends it with an ERROR line and a TimeoutException - that is how the tool reports "nothing arrived", not a failure of the cluster. That default surprises everyone once. --from-beginning starts at the oldest record still in the log - only if the group has no committed offset yet:
$ kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic notifications --from-beginning --max-messages 3
The consumer rebalance protocol (KIP-848) is production-ready! Set group.protocol=consumer to try it out. See https://kafka.apache.org/documentation/#consumer_rebalance_protocol
{"to":"c-1000","template":"order_confirmed","order_id":"o-100000","ts":"2026-09-21T08:00:03.000Z"}
{"to":"c-1049","template":"invoice_ready","order_id":"o-100007","ts":"2026-09-21T08:00:03.250Z"}
{"to":"c-1098","template":"shipped","order_id":"o-100014","ts":"2026-09-21T08:00:03.500Z"}
Processed a total of 3 messages
The first line on stderr is the tool advertising the new consumer group protocol (KIP-848, next lesson); the console consumer prints it whenever you use the default "classic" protocol.
What a record carries besides the value comes out with --formatter-property (--property in older guides, now deprecated):
$ kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic notifications --from-beginning --max-messages 2 --formatter-property print.timestamp=true --formatter-property print.partition=true --formatter-property print.offset=true --formatter-property print.key=true
The consumer rebalance protocol (KIP-848) is production-ready! Set group.protocol=consumer to try it out. See https://kafka.apache.org/documentation/#consumer_rebalance_protocol
CreateTime:1789977603000 Partition:0 Offset:0 c-1000 {"to":"c-1000","template":"order_confirmed","order_id":"o-100000","ts":"2026-09-21T08:00:03.000Z"}
CreateTime:1789977603250 Partition:0 Offset:1 c-1049 {"to":"c-1049","template":"invoice_ready","order_id":"o-100007","ts":"2026-09-21T08:00:03.250Z"}
Processed a total of 2 messages
The fields come in a fixed order - timestamp, partition, offset, key, value - separated by a TAB (key.separator changes it). CreateTime: is the producer's timestamp; a topic with message.timestamp.type=LogAppendTime shows the broker's arrival time instead.
To read one partition from one offset, give both:
$ kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic notifications --partition 0 --offset earliest --max-messages 2 --formatter-property print.offset=true
Offset:0 {"to":"c-1000","template":"order_confirmed","order_id":"o-100000","ts":"2026-09-21T08:00:03.000Z"}
Offset:1 {"to":"c-1049","template":"invoice_ready","order_id":"o-100007","ts":"2026-09-21T08:00:03.250Z"}
Processed a total of 2 messages
--partition cannot be combined with --group: reading by hand from a fixed position is the opposite of group consumption.
The console consumer is a group member too
Without --group, the consumer joins a throw-away group called console-consumer-<random number> with automatic commits turned off. With --group billing-debug it joins (or creates) that group and commits what it read. Never point a debugging consumer at a production group's name: it takes partitions away from the real consumers (a rebalance), and its commits move the group's position.
Where are the ends?
kafka-get-offsets.sh asks each partition's leader for an offset:
$ kafka-get-offsets.sh --bootstrap-server kafka-1:9092 --topic notifications
notifications:0:518402
notifications:1:518402
notifications:2:518401
$ kafka-get-offsets.sh --bootstrap-server kafka-1:9092 --topic notifications --time earliest
notifications:0:0
notifications:1:0
notifications:2:0
The default (--time latest, -1) is the end of the log (the high watermark: the next offset to be written); earliest (-2) is the first offset still on disk - retention moves it forward. The difference is how many records each partition holds right now. A timestamp in milliseconds instead returns the first offset at or after that time, which is exactly what an offset reset to a date uses (lesson 7).
In an interview: "A new consumer starts at the end unless the group has a committed offset or you ask for earliest. A keyed record always lands in the same partition, which is what gives per-key ordering."