What a partition looks like on disk
Every replica is a directory <topic>-<partition> under log.dirs. Kafka does not keep one huge file per partition: the log is cut into segments, and each segment is a set of files named after the first offset it contains:
$ ssh kafka-1 ls -l /var/lib/kafka/data/clickstream-0/
total 1716056
-rw-r--r-- 1 kafka kafka 204800 Sep 21 21:27 00000000000000000000.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 21 21:27 00000000000000000000.log
-rw-r--r-- 1 kafka kafka 10 Sep 21 21:27 00000000000000000000.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 21 21:27 00000000000000000000.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 21 22:54 00000000000000262144.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 21 22:54 00000000000000262144.log
-rw-r--r-- 1 kafka kafka 10 Sep 21 22:54 00000000000000262144.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 21 22:54 00000000000000262144.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 00:22 00000000000000524288.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 00:22 00000000000000524288.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 00:22 00000000000000524288.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 00:22 00000000000000524288.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 01:49 00000000000000786432.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 01:49 00000000000000786432.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 01:49 00000000000000786432.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 01:49 00000000000000786432.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 03:17 00000000000001048576.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 03:17 00000000000001048576.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 03:17 00000000000001048576.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 03:17 00000000000001048576.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 04:44 00000000000001310720.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 04:44 00000000000001310720.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 04:44 00000000000001310720.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 04:44 00000000000001310720.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 06:11 00000000000001572864.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 06:11 00000000000001572864.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 06:11 00000000000001572864.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 06:11 00000000000001572864.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 07:39 00000000000001835008.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 07:39 00000000000001835008.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 07:39 00000000000001835008.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 07:39 00000000000001835008.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 09:06 00000000000002097152.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 09:06 00000000000002097152.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 09:06 00000000000002097152.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 09:06 00000000000002097152.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 10:33 00000000000002359296.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 10:33 00000000000002359296.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 10:33 00000000000002359296.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 10:33 00000000000002359296.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 12:01 00000000000002621440.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 12:01 00000000000002621440.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 12:01 00000000000002621440.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 12:01 00000000000002621440.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 13:28 00000000000002883584.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 13:28 00000000000002883584.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 13:28 00000000000002883584.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 13:28 00000000000002883584.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 14:56 00000000000003145728.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 14:56 00000000000003145728.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 14:56 00000000000003145728.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 14:56 00000000000003145728.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 16:23 00000000000003407872.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 16:23 00000000000003407872.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 16:23 00000000000003407872.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 16:23 00000000000003407872.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 17:50 00000000000003670016.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 17:50 00000000000003670016.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 17:50 00000000000003670016.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 17:50 00000000000003670016.timeindex
-rw-r--r-- 1 kafka kafka 204800 Sep 22 19:18 00000000000003932160.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 19:18 00000000000003932160.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 19:18 00000000000003932160.snapshot
-rw-r--r-- 1 kafka kafka 307200 Sep 22 19:18 00000000000003932160.timeindex
-rw-r--r-- 1 kafka kafka 10485760 Sep 22 20:00 00000000000004194304.index
-rw-r--r-- 1 kafka kafka 50278400 Sep 22 20:00 00000000000004194304.log
-rw-r--r-- 1 kafka kafka 10 Sep 22 20:00 00000000000004194304.snapshot
-rw-r--r-- 1 kafka kafka 10485756 Sep 22 20:00 00000000000004194304.timeindex
-rw-r--r-- 1 kafka kafka 8 Sep 22 20:00 leader-epoch-checkpoint
-rw-r--r-- 1 kafka kafka 44 Sep 22 20:00 partition.metadata
.log- the records themselves, appended in batches..index- offset -> byte position in the.log, one entry everyindex.interval.bytes(4 KiB)..timeindex- timestamp -> offset, used for "give me the offset at 08:00"..snapshot- producer state (for idempotence) at the segment boundary.leader-epoch-checkpoint,partition.metadata- which leader epoch started at which offset, and the topic id.
The last segment is the active one: the only file being written. Its index files are created at full size (segment.index.bytes, 10 MiB) and trimmed when it closes - which is why ls -l shows 10485760 bytes for an almost empty index. A segment closes ("rolls") when it reaches segment.bytes (1 GiB by default) or when it is older than segment.ms (7 days).
In the data directory itself, next to the partition directories, are the broker's checkpoint files and its meta.properties:
$ ssh kafka-1 ls /var/lib/kafka/data | head -12
__consumer_offsets-0
__consumer_offsets-1
__consumer_offsets-10
__consumer_offsets-11
__consumer_offsets-12
__consumer_offsets-13
__consumer_offsets-14
__consumer_offsets-15
__consumer_offsets-16
__consumer_offsets-17
__consumer_offsets-18
__consumer_offsets-19
$ ssh kafka-1 df -h /var/lib/kafka
Filesystem Size Used Avail Use% Mounted on
/dev/vdb1 49G 13G 34G 27% /var/lib/kafka
$ ssh kafka-1 du -sh /var/lib/kafka/data/clickstream-0 /var/lib/kafka/data/orders-0
1.7G /var/lib/kafka/data/clickstream-0
597M /var/lib/kafka/data/orders-0
Retention: deleting whole segments
cleanup.policy=delete (the default) removes old data one closed segment at a time:
retention.ms(default 7 days, from the broker'slog.retention.hours=168): a closed segment is deleted when its newest record is older than this.retention.bytes(default -1, no limit): while the partition is bigger than this, the oldest closed segment goes. It is per partition: 10 GB on a topic with 12 partitions and RF 3 is up to 360 GB of disk across the cluster.
The check runs every log.retention.check.interval.ms (5 minutes by default; the lab runs it every minute so you see the effect). Two consequences that surprise people:
- The active segment is never deleted. A topic that receives 100 MB a day with 1 GiB segments and
retention.msof one hour still keeps ten days of data: the segment never fills, so it never closes.segment.ms(or a smallersegment.bytes) makes it roll. - The log start moves in steps.
kafka-get-offsets --time earliestjumps forward by a segment at a time, and a consumer that was behind the new log start finds its offset gone (auto.offset.resetagain).
clickstream has a day of data in 100 MiB segments. Give it one hour of retention:
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --entity-type topics --entity-name clickstream --add-config retention.ms=3600000
Completed updating config for topic clickstream.
$ ssh kafka-1 "grep -m2 'due to log retention time' /var/log/kafka/server.log"
[2026-09-22 20:01:03,680] INFO [UnifiedLog partition=clickstream-2, dir=/var/lib/kafka/data] Deleting segment LogSegment(baseOffset=0, size=104857600, lastModifiedTime=1790026049880, largestRecordTimestamp=1790026049880) due to log retention time 3600000ms breach based on the largest record timestamp in the segment (org.apache.kafka.storage.internals.log.UnifiedLog)
[2026-09-22 20:01:03,680] INFO [UnifiedLog partition=clickstream-2, dir=/var/lib/kafka/data] Deleting segment LogSegment(baseOffset=262144, size=104857600, lastModifiedTime=1790031292760, largestRecordTimestamp=1790031292760) due to log retention time 3600000ms breach based on the largest record timestamp in the segment (org.apache.kafka.storage.internals.log.UnifiedLog)
$ ssh kafka-1 du -sh /var/lib/kafka/data/clickstream-0
169M /var/lib/kafka/data/clickstream-0
$ kafka-get-offsets.sh --bootstrap-server kafka-1:9092 --topic clickstream --time earliest
clickstream:0:3932160
clickstream:1:3932160
clickstream:2:3932160
The UnifiedLog lines name each deleted segment with its base offset, size and largest timestamp, then "Incremented log start offset". Followers delete the same segments when they see the leader's new log start.
Compaction: keep the last value per key
cleanup.policy=compact keeps, for every key, the newest record and drops older ones - a table of "current state" (customer profiles, configuration, the __consumer_offsets topic itself). The log cleaner threads rewrite closed segments when the share of "dirty" (superseded) records passes min.cleanable.dirty.ratio (0.5). A record with a key and a null value is a tombstone: it deletes the key, and is itself removed after delete.retention.ms (1 day). The console producer writes null with the reader property null.marker. Compaction works on keys, so records without a key cannot go to a compacted topic.
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --create --topic customer-profiles --replica-assignment 1:2:3 --config cleanup.policy=compact --config segment.ms=20000 --config min.cleanable.dirty.ratio=0.01
Created topic customer-profiles.
$ printf 'c-1:gold\nc-2:free\nc-1:platinum\nc-3:free\nc-2:NULL\n' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic customer-profiles --reader-property parse.key=true --reader-property key.separator=: --reader-property null.marker=NULL
$ printf 'c-4:gold\n' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic customer-profiles --reader-property parse.key=true --reader-property key.separator=:
$ kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic customer-profiles --from-beginning --timeout-ms 5000 --formatter-property print.key=true --formatter-property print.offset=true
The consumer rebalance protocol (KIP-848) is production-ready! Set group.protocol=consumer to try it out. See https://kafka.apache.org/documentation/#consumer_rebalance_protocol
Offset:2 c-1 platinum
Offset:3 c-3 free
Offset:4 c-2 null
Offset:5 c-4 gold
[2026-09-22 20:02:59,300] ERROR Error processing message, terminating consumer process: (org.apache.kafka.tools.consumer.ConsoleConsumer)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 4 messages
$ ssh kafka-1 "grep -E 'Cleaning log|cleaned log|fewer messages' /var/log/kafka/log-cleaner.log"
[2026-09-22 20:02:03,680] INFO Cleaner 0: Cleaning log customer-profiles-0 (cleaning prior to Tue Sep 22 20:01:19 UTC 2026, discarding tombstones prior to upper bound deletion horizon Mon Sep 21 20:02:03 UTC 2026)... (org.apache.kafka.storage.internals.log.Cleaner)
Log cleaner thread 0 cleaned log customer-profiles-0 (dirty section = [0, 5])
40.0% size reduction (40.0% fewer messages)
(--replica-assignment 1:2:3 placed the single partition by hand, broker 1 first, so its leader - the broker whose cleaner does the work - is kafka-1.) c-1:gold (offset 0) and c-2:free (offset 1) are gone - newer values replaced them; c-2 still shows its tombstone (printed as null) until delete.retention.ms passes. Offsets are never renumbered: there are now holes. The cleaner logs to its own file, log-cleaner.log. (simulator: the lab's cleaner works on records written by hand; the generated history of the other topics is not compacted.)
A full disk
When a broker cannot write to its log directory, Kafka marks the directory offline. If it is the only directory - or the one holding the metadata log, as here, where metadata.log.dir defaults to the first log.dirs entry - the broker stops at once:
ERROR Error while appending records to audit-debug-3 in dir /var/lib/kafka/data (org.apache.kafka.storage.internals.log.LogDirFailureChannel)
java.io.IOException: No space left on device
WARN [ReplicaManager broker=3] Stopping serving replicas in dir /var/lib/kafka/data with uuid ... because the log directory has failed.
ERROR [ReplicaManager broker=3] Shutdown broker because the metadata log dir /var/lib/kafka/data has failed
and the other brokers, which hold the same replicas, are usually only a little behind. Restarting does not help while the disk is full. What does: free space that is not Kafka data (old heap dumps, logs, a ballast file kept for this moment), start the broker, then lower retention for whatever filled it. Never delete segment files by hand on a running broker: the index and the checkpoints still point at them.
kafka-log-dirs.sh shows the size of every replica from the cluster's side, without logging into the brokers:
$ kafka-log-dirs.sh --bootstrap-server kafka-1:9092 --describe --broker-list 1 --topic-list clickstream
Querying brokers for log directories information
Received log directory information from brokers 1
{"brokers":[{"broker":1,"logDirs":[{"partitions":[{"partition":"clickstream-0","size":155136000,"offsetLag":0,"isFuture":false},{"partition":"clickstream-1","size":155136000,"offsetLag":0,"isFuture":false},{"partition":"clickstream-2","size":155136000,"offsetLag":0,"isFuture":false}],"error":null,"logDir":"/var/lib/kafka/data"}]}],"version":1}
In an interview: "Retention deletes whole closed segments by age or size; retention.bytes is per partition, and the active segment is never deleted, so a quiet topic can keep data far past retention.ms. Compaction keeps the latest value per key, tombstones delete keys."