OnCallReadyStart learning

Runtimes and data Lesson 28.15 · Kafka Operations · 18 min read

Storage: segments, retention, compaction and full disks

In plain words

Each partition is kept as a stack of notebooks. Only the top notebook is being written; when it is full or old enough, it is closed and a new one started. Retention throws away whole old notebooks from the bottom of the stack - never single pages, and never the notebook still being written. Compaction instead rewrites old notebooks keeping only the newest line for each name.

What a partition looks like on disk

Every replica is a directory <topic>-<partition> under log.dirs. Kafka does not keep one huge file per partition: the log is cut into segments, and each segment is a set of files named after the first offset it contains:

$ ssh kafka-1 ls -l /var/lib/kafka/data/clickstream-0/
total 1716056
-rw-r--r-- 1 kafka kafka    204800 Sep 21 21:27 00000000000000000000.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 21 21:27 00000000000000000000.log
-rw-r--r-- 1 kafka kafka        10 Sep 21 21:27 00000000000000000000.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 21 21:27 00000000000000000000.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 21 22:54 00000000000000262144.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 21 22:54 00000000000000262144.log
-rw-r--r-- 1 kafka kafka        10 Sep 21 22:54 00000000000000262144.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 21 22:54 00000000000000262144.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 00:22 00000000000000524288.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 00:22 00000000000000524288.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 00:22 00000000000000524288.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 00:22 00000000000000524288.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 01:49 00000000000000786432.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 01:49 00000000000000786432.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 01:49 00000000000000786432.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 01:49 00000000000000786432.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 03:17 00000000000001048576.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 03:17 00000000000001048576.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 03:17 00000000000001048576.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 03:17 00000000000001048576.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 04:44 00000000000001310720.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 04:44 00000000000001310720.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 04:44 00000000000001310720.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 04:44 00000000000001310720.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 06:11 00000000000001572864.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 06:11 00000000000001572864.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 06:11 00000000000001572864.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 06:11 00000000000001572864.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 07:39 00000000000001835008.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 07:39 00000000000001835008.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 07:39 00000000000001835008.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 07:39 00000000000001835008.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 09:06 00000000000002097152.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 09:06 00000000000002097152.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 09:06 00000000000002097152.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 09:06 00000000000002097152.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 10:33 00000000000002359296.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 10:33 00000000000002359296.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 10:33 00000000000002359296.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 10:33 00000000000002359296.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 12:01 00000000000002621440.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 12:01 00000000000002621440.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 12:01 00000000000002621440.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 12:01 00000000000002621440.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 13:28 00000000000002883584.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 13:28 00000000000002883584.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 13:28 00000000000002883584.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 13:28 00000000000002883584.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 14:56 00000000000003145728.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 14:56 00000000000003145728.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 14:56 00000000000003145728.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 14:56 00000000000003145728.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 16:23 00000000000003407872.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 16:23 00000000000003407872.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 16:23 00000000000003407872.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 16:23 00000000000003407872.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 17:50 00000000000003670016.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 17:50 00000000000003670016.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 17:50 00000000000003670016.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 17:50 00000000000003670016.timeindex
-rw-r--r-- 1 kafka kafka    204800 Sep 22 19:18 00000000000003932160.index
-rw-r--r-- 1 kafka kafka 104857600 Sep 22 19:18 00000000000003932160.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 19:18 00000000000003932160.snapshot
-rw-r--r-- 1 kafka kafka    307200 Sep 22 19:18 00000000000003932160.timeindex
-rw-r--r-- 1 kafka kafka  10485760 Sep 22 20:00 00000000000004194304.index
-rw-r--r-- 1 kafka kafka  50278400 Sep 22 20:00 00000000000004194304.log
-rw-r--r-- 1 kafka kafka        10 Sep 22 20:00 00000000000004194304.snapshot
-rw-r--r-- 1 kafka kafka  10485756 Sep 22 20:00 00000000000004194304.timeindex
-rw-r--r-- 1 kafka kafka         8 Sep 22 20:00 leader-epoch-checkpoint
-rw-r--r-- 1 kafka kafka        44 Sep 22 20:00 partition.metadata
  • .log - the records themselves, appended in batches.
  • .index - offset -> byte position in the .log, one entry every index.interval.bytes (4 KiB).
  • .timeindex - timestamp -> offset, used for "give me the offset at 08:00".
  • .snapshot - producer state (for idempotence) at the segment boundary.
  • leader-epoch-checkpoint, partition.metadata - which leader epoch started at which offset, and the topic id.

The last segment is the active one: the only file being written. Its index files are created at full size (segment.index.bytes, 10 MiB) and trimmed when it closes - which is why ls -l shows 10485760 bytes for an almost empty index. A segment closes ("rolls") when it reaches segment.bytes (1 GiB by default) or when it is older than segment.ms (7 days).

In the data directory itself, next to the partition directories, are the broker's checkpoint files and its meta.properties:

$ ssh kafka-1 ls /var/lib/kafka/data | head -12
__consumer_offsets-0
__consumer_offsets-1
__consumer_offsets-10
__consumer_offsets-11
__consumer_offsets-12
__consumer_offsets-13
__consumer_offsets-14
__consumer_offsets-15
__consumer_offsets-16
__consumer_offsets-17
__consumer_offsets-18
__consumer_offsets-19
$ ssh kafka-1 df -h /var/lib/kafka
Filesystem  Size  Used Avail Use% Mounted on
/dev/vdb1    49G   13G   34G  27% /var/lib/kafka
$ ssh kafka-1 du -sh /var/lib/kafka/data/clickstream-0 /var/lib/kafka/data/orders-0
1.7G	/var/lib/kafka/data/clickstream-0
597M	/var/lib/kafka/data/orders-0

Retention: deleting whole segments

cleanup.policy=delete (the default) removes old data one closed segment at a time:

  • retention.ms (default 7 days, from the broker's log.retention.hours=168): a closed segment is deleted when its newest record is older than this.
  • retention.bytes (default -1, no limit): while the partition is bigger than this, the oldest closed segment goes. It is per partition: 10 GB on a topic with 12 partitions and RF 3 is up to 360 GB of disk across the cluster.

The check runs every log.retention.check.interval.ms (5 minutes by default; the lab runs it every minute so you see the effect). Two consequences that surprise people:

  1. The active segment is never deleted. A topic that receives 100 MB a day with 1 GiB segments and retention.ms of one hour still keeps ten days of data: the segment never fills, so it never closes. segment.ms (or a smaller segment.bytes) makes it roll.
  2. The log start moves in steps. kafka-get-offsets --time earliest jumps forward by a segment at a time, and a consumer that was behind the new log start finds its offset gone (auto.offset.reset again).

clickstream has a day of data in 100 MiB segments. Give it one hour of retention:

$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --entity-type topics --entity-name clickstream --add-config retention.ms=3600000
Completed updating config for topic clickstream.
$ ssh kafka-1 "grep -m2 'due to log retention time' /var/log/kafka/server.log"
[2026-09-22 20:01:03,680] INFO [UnifiedLog partition=clickstream-2, dir=/var/lib/kafka/data] Deleting segment LogSegment(baseOffset=0, size=104857600, lastModifiedTime=1790026049880, largestRecordTimestamp=1790026049880) due to log retention time 3600000ms breach based on the largest record timestamp in the segment (org.apache.kafka.storage.internals.log.UnifiedLog)
[2026-09-22 20:01:03,680] INFO [UnifiedLog partition=clickstream-2, dir=/var/lib/kafka/data] Deleting segment LogSegment(baseOffset=262144, size=104857600, lastModifiedTime=1790031292760, largestRecordTimestamp=1790031292760) due to log retention time 3600000ms breach based on the largest record timestamp in the segment (org.apache.kafka.storage.internals.log.UnifiedLog)
$ ssh kafka-1 du -sh /var/lib/kafka/data/clickstream-0
169M	/var/lib/kafka/data/clickstream-0
$ kafka-get-offsets.sh --bootstrap-server kafka-1:9092 --topic clickstream --time earliest
clickstream:0:3932160
clickstream:1:3932160
clickstream:2:3932160

The UnifiedLog lines name each deleted segment with its base offset, size and largest timestamp, then "Incremented log start offset". Followers delete the same segments when they see the leader's new log start.

Compaction: keep the last value per key

cleanup.policy=compact keeps, for every key, the newest record and drops older ones - a table of "current state" (customer profiles, configuration, the __consumer_offsets topic itself). The log cleaner threads rewrite closed segments when the share of "dirty" (superseded) records passes min.cleanable.dirty.ratio (0.5). A record with a key and a null value is a tombstone: it deletes the key, and is itself removed after delete.retention.ms (1 day). The console producer writes null with the reader property null.marker. Compaction works on keys, so records without a key cannot go to a compacted topic.

$ kafka-topics.sh --bootstrap-server kafka-1:9092 --create --topic customer-profiles --replica-assignment 1:2:3 --config cleanup.policy=compact --config segment.ms=20000 --config min.cleanable.dirty.ratio=0.01
Created topic customer-profiles.
$ printf 'c-1:gold\nc-2:free\nc-1:platinum\nc-3:free\nc-2:NULL\n' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic customer-profiles --reader-property parse.key=true --reader-property key.separator=: --reader-property null.marker=NULL
$ printf 'c-4:gold\n' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic customer-profiles --reader-property parse.key=true --reader-property key.separator=:
$ kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic customer-profiles --from-beginning --timeout-ms 5000 --formatter-property print.key=true --formatter-property print.offset=true
The consumer rebalance protocol (KIP-848) is production-ready! Set group.protocol=consumer to try it out. See https://kafka.apache.org/documentation/#consumer_rebalance_protocol
Offset:2	c-1	platinum
Offset:3	c-3	free
Offset:4	c-2	null
Offset:5	c-4	gold
[2026-09-22 20:02:59,300] ERROR Error processing message, terminating consumer process:  (org.apache.kafka.tools.consumer.ConsoleConsumer)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 4 messages
$ ssh kafka-1 "grep -E 'Cleaning log|cleaned log|fewer messages' /var/log/kafka/log-cleaner.log"
[2026-09-22 20:02:03,680] INFO Cleaner 0: Cleaning log customer-profiles-0 (cleaning prior to Tue Sep 22 20:01:19 UTC 2026, discarding tombstones prior to upper bound deletion horizon Mon Sep 21 20:02:03 UTC 2026)... (org.apache.kafka.storage.internals.log.Cleaner)
	Log cleaner thread 0 cleaned log customer-profiles-0 (dirty section = [0, 5])
	40.0% size reduction (40.0% fewer messages)

(--replica-assignment 1:2:3 placed the single partition by hand, broker 1 first, so its leader - the broker whose cleaner does the work - is kafka-1.) c-1:gold (offset 0) and c-2:free (offset 1) are gone - newer values replaced them; c-2 still shows its tombstone (printed as null) until delete.retention.ms passes. Offsets are never renumbered: there are now holes. The cleaner logs to its own file, log-cleaner.log. (simulator: the lab's cleaner works on records written by hand; the generated history of the other topics is not compacted.)

A full disk

When a broker cannot write to its log directory, Kafka marks the directory offline. If it is the only directory - or the one holding the metadata log, as here, where metadata.log.dir defaults to the first log.dirs entry - the broker stops at once:

ERROR Error while appending records to audit-debug-3 in dir /var/lib/kafka/data (org.apache.kafka.storage.internals.log.LogDirFailureChannel)
java.io.IOException: No space left on device
WARN [ReplicaManager broker=3] Stopping serving replicas in dir /var/lib/kafka/data with uuid ... because the log directory has failed.
ERROR [ReplicaManager broker=3] Shutdown broker because the metadata log dir /var/lib/kafka/data has failed

and the other brokers, which hold the same replicas, are usually only a little behind. Restarting does not help while the disk is full. What does: free space that is not Kafka data (old heap dumps, logs, a ballast file kept for this moment), start the broker, then lower retention for whatever filled it. Never delete segment files by hand on a running broker: the index and the checkpoints still point at them.

kafka-log-dirs.sh shows the size of every replica from the cluster's side, without logging into the brokers:

$ kafka-log-dirs.sh --bootstrap-server kafka-1:9092 --describe --broker-list 1 --topic-list clickstream
Querying brokers for log directories information
Received log directory information from brokers 1
{"brokers":[{"broker":1,"logDirs":[{"partitions":[{"partition":"clickstream-0","size":155136000,"offsetLag":0,"isFuture":false},{"partition":"clickstream-1","size":155136000,"offsetLag":0,"isFuture":false},{"partition":"clickstream-2","size":155136000,"offsetLag":0,"isFuture":false}],"error":null,"logDir":"/var/lib/kafka/data"}]}],"version":1}

In an interview: "Retention deletes whole closed segments by age or size; retention.bytes is per partition, and the active segment is never deleted, so a quiet topic can keep data far past retention.ms. Compaction keeps the latest value per key, tombstones delete keys."

Why it helps

Disk usage is the slowest-moving Kafka incident and one of the most damaging: a full log directory stops the broker. Understanding segments, retention.ms and retention.bytes per partition, and the active segment rule explains why data stays longer than expected and how much disk a topic really needs. It also tells you what is safe to delete when a disk is full.

Commands in this lesson

ssh kafka-configs.sh kafka-get-offsets.sh kafka-topics.sh printf kafka-console-consumer.sh kafka-log-dirs.sh

FAQ

What are the files in a partition directory?

Each segment has a .log with the records, an .index from offset to position, a .timeindex from timestamp to offset, and a .snapshot of producer state. The directory also holds leader-epoch-checkpoint and partition.metadata with the topic id.

Why does a topic keep data longer than retention.ms?

Retention deletes only closed segments, and the active segment is never deleted. A topic that does not fill its segment (1 GiB by default) keeps it until segment.ms rolls it, which is 7 days by default, whatever retention.ms says.

Is retention.bytes for the whole topic?

No, it is per partition, and every replica holds that much on its broker. 10 GB on a topic with 12 partitions and replication factor 3 can use up to 360 GB of disk in the cluster.

What is a tombstone?

A record with a key and a null value in a compacted topic. It tells the log cleaner to delete that key; the tombstone itself is removed after delete.retention.ms (one day by default), so consumers that read slowly still see the deletion.

Can I delete segment files to free space?

Not on a running broker: its index and checkpoint files still point at them. Free space that is not Kafka data first (heap dumps, old logs), start the broker, and lower retention so Kafka deletes the segments itself.

In an interview Mid

How does Kafka retention work, and why might a topic use more disk than you expect?

The log is split into segments and retention deletes whole closed segments when their newest record is older than retention.ms or while the partition exceeds retention.bytes, checked every log.retention.check.interval.ms. retention.bytes is per partition and per replica, and the active segment is never deleted, so a quiet topic keeps data until segment.ms rolls it. Compaction keeps the latest value per key instead.

Also asked: What is the difference between delete and compact? · What happens when a broker's data disk is full? · How do you size retention.bytes for a disk budget?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.