OnCallReadyStart learning

Kafka Operations: glossary

60 terms, in plain words, with where each one is taught.

acks (acks=all, acks=1, request.required.acks)
The producer setting for how many acknowledgements a write needs: 0 none, 1 the leader, all (-1) every ISR member. all is the default since Kafka 3.0. Lesson 28.7
Apache Kafka (Kafka)
A distributed, replicated commit log: producers append records to topics, consumers read them at their own pace, and the records stay until retention removes them. Lesson 28.1
auto.offset.reset (earliest, latest)
Where a consumer starts when its group has no committed offset (or it expired): latest (the default, the end) or earliest (the oldest record still kept). Lesson 28.9
broker (Kafka broker)
A Kafka server that stores partitions and serves producers and consumers (process.roles=broker). In the lab: kafka-1 to kafka-3, node ids 1-3. Lesson 28.1
consumer group (group.id)
Consumers that share a group id and split a topic's partitions between them, each partition to one member, committing their offsets under the group. Lesson 28.11
consumer lag (LAG, lag)
LOG-END-OFFSET minus the committed CURRENT-OFFSET, per partition: how many records the group has not processed yet. Lesson 28.11
controlled shutdown (clean shutdown)
What a broker does on SIGTERM: asks the controller to move its leaderships away, leaves the ISRs, closes its logs cleanly so the next start needs no recovery. Lesson 28.7
controller (KRaft controller, active controller)
A Kafka node with process.roles=controller: a voter in the metadata quorum. The active controller (the Raft leader) decides topic placement, leader elections and broker fencing. Lesson 28.3
dynamic broker config (DYNAMIC_BROKER_CONFIG, DYNAMIC_DEFAULT_BROKER_CONFIG)
A broker setting stored in the cluster metadata by kafka-configs, applied without a restart; per broker or cluster-wide. Read-only configs (log.dirs, node.id) can only be static. Lesson 28.17
ELR (eligible leader replicas, Elr:, LastKnownElr)
Replicas removed from the ISR while it was at or below min.insync.replicas but known to have every committed record, so they may still become leader cleanly. On by default for new clusters since Kafka 4.1. Lesson 28.7
fenced broker (fencing, isFenced)
A broker the controller does not treat as available (not yet caught up after start, or its session timed out); it leads nothing and is out of the ISRs. Lesson 28.7
group coordinator (__consumer_offsets, COORDINATOR)
The broker that leads the __consumer_offsets partition a group id hashes to; it tracks the members, runs rebalances and stores committed offsets. Lesson 28.11
high watermark (HW)
The offset up to which every ISR member has the records; consumers only read below it, so they never see a record that could still vanish. Lesson 28.7
idempotent producer (enable.idempotence)
A producer whose retried batches the broker de-duplicates by producer id and sequence number, so a retry does not write a duplicate. On by default since Kafka 3.0. Lesson 28.7
INCONSISTENT_CLUSTER_ID (inconsistent cluster id)
The controllers' answer to a node formatted for another cluster: its fetches of the metadata log are refused (an ERROR line many times a second), so it never registers and never leaves STARTING. Lesson 28.3
ISR (in-sync replicas, Isr:)
The replicas that are caught up with the leader. Only they count for acks=all and only they may become leader cleanly. A follower behind for replica.lag.time.max.ms (30 s) is removed. Lesson 28.7
KAFKA_HEAP_OPTS (broker heap, ExitOnOutOfMemoryError)
The environment variable with the broker JVM's heap flags (-Xmx/-Xms); the rest of the memory is left to the page cache, which serves most reads. Lesson 28.23
kafka-configs.sh (--entity-default, --add-config, --delete-config)
The tool for dynamic configs: topic overrides, per-broker and cluster-wide broker configs; --describe --all shows every value with its source and synonyms. Lesson 28.17
kafka-console-consumer.sh (console consumer, --from-beginning, print.key)
The tool that prints a topic's records; starts at the end unless --from-beginning, stops with --max-messages or --timeout-ms, shows key, partition, offset and timestamp with --formatter-property. Lesson 28.9
kafka-console-producer.sh (console producer, parse.key, key.separator)
The tool that turns standard input into records: one line per record; with parse.key=true each line is split at the key separator into key and value. Lesson 28.9
kafka-consumer-groups.sh (kafka-consumer-groups --describe, --members, --state)
The tool to list groups, describe their offsets and lag, members and state, reset or delete offsets, and delete groups. Lesson 28.11
kafka-get-offsets.sh (get offsets, --time earliest)
Prints topic:partition:offset for the end of each partition (latest), the first offset still kept (earliest) or the first at a timestamp. Lesson 28.9
kafka-jmx.sh (JMX, JMX_PORT)
The tool that reads JMX metrics from a broker or controller by object name, once (--one-time true) or periodically. Lesson 28.23
kafka-log-dirs.sh (log-dirs --describe)
Prints, as JSON, every log directory of the chosen brokers with the size and offset lag of each replica in it. Lesson 28.15
kafka-metadata-quorum.sh (describe --status, describe --replication)
The tool that shows the metadata quorum: --status for the leader, epoch and follower lag, --replication for every voter and observer's log end offset and lag. Lesson 28.3
kafka-storage.sh (kafka-storage format, random-uuid)
The tool that formats a node's log directories (writes meta.properties with the cluster id, node id and directory id), shows them (info) and makes cluster ids (random-uuid). Lesson 28.3
kafka-topics.sh (kafka-topics --describe, --under-replicated-partitions, --under-min-isr-partitions)
The tool to list, create, describe, alter (add partitions) and delete topics; --describe prints Leader, Replicas, Isr and Elr per partition, and the report flags show only the problem partitions. Lesson 28.5
KIP (Kafka Improvement Proposal, KIPs)
A Kafka Improvement Proposal: the numbered design document every user-visible Kafka change goes through (KIP-500 removed ZooKeeper, KIP-848 brought the new consumer group protocol). Release notes and docs name features by their KIP. Lesson 28.11
KIP-848 (group.protocol=consumer, consumer rebalance protocol)
The new consumer group protocol (production-ready in Kafka 4.0): the broker computes assignments and rebalances incrementally. Clients opt in with group.protocol=consumer. Lesson 28.11
KRaft (KRaft mode, Kafka Raft)
Kafka's built-in metadata quorum, based on Raft: controllers replicate the __cluster_metadata log. The only mode since Kafka 4.0 removed ZooKeeper. Lesson 28.3
log compaction (cleanup.policy=compact, log cleaner, tombstone)
Keeping only the newest record per key in closed segments instead of deleting by age; a key with a null value (a tombstone) deletes the key. Lesson 28.15
log segment (segment, .log file, active segment)
One file set (.log, .index, .timeindex) of a partition, named after its first offset. Only the last, active segment is written; retention deletes whole closed segments. Lesson 28.15
max.poll.interval.ms (max.poll.records, poll timeout)
How long a consumer may take between poll() calls (5 min by default) before it is kicked out of the group. max.poll.records x processing time must fit in it, or the group rebalances for ever. Lesson 28.11
meta.properties (cluster.id, directory.id)
The file in every Kafka log directory that ties it to a cluster and a node. A broker whose node.id or cluster.id does not match refuses to start or to register. Lesson 28.3
metadata quorum (__cluster_metadata, quorum)
The controllers that replicate the cluster metadata log. A majority must be up to elect a leader and commit changes: 3 voters survive 1 failure, 5 survive 2. Lesson 28.3
min.insync.replicas (min ISR)
The broker/topic floor for acks=all: with fewer replicas in the ISR, the leader refuses the write with NOT_ENOUGH_REPLICAS. RF 3 with 2 tolerates one broker down. Lesson 28.7
murmur2 partitioner (default partitioner, sticky partitioner)
How a keyed record picks its partition: murmur2(key bytes), made positive, modulo the partition count. Records without a key go in sticky batches to one partition at a time. Lesson 28.9
NOT_ENOUGH_REPLICAS (NotEnoughReplicasException)
The error an acks=all producer gets when the partition's ISR is below min.insync.replicas. The fix is the brokers, not a lower min.insync.replicas. Lesson 28.7
OfflinePartitionsCount (ActiveControllerCount, UnderMinIsrPartitionCount)
Alerting metrics: partitions without a leader (on the active controller), the controller that is active (exactly one should say 1), partitions below min.insync.replicas (on brokers). Lesson 28.23
offset (Kafka offset)
The position of a record in its partition. Offsets only grow and are never reused; compaction and retention leave gaps rather than renumbering. Lesson 28.1
offset reset (--reset-offsets, --to-datetime, --shift-by)
Moving a consumer group's committed offsets with kafka-consumer-groups: replay, skip or recover. The group must be Empty; without --execute it is a dry run. Lesson 28.13
offsets.retention.minutes (offset expiry)
How long the committed offsets of an Empty group are kept (7 days); afterwards the group starts where auto.offset.reset says. Lesson 28.13
partition (topic partition)
One ordered, append-only log of a topic, with offsets 0, 1, 2... Ordering is guaranteed inside a partition only; it is also the unit of parallelism and of replication. Lesson 28.1
partition leader (Leader:)
The replica that serves a partition's producers and consumers. The controller elects it from the ISR. Lesson 28.5
partition reassignment (kafka-reassign-partitions.sh, --generate, --verify)
Changing which brokers hold a partition's replicas: new replicas copy the data and join the ISR, then the removed ones leave. --verify reports progress and removes the throttles. Lesson 28.19
preferred leader (preferred replica)
The first broker in a partition's replica list; Kafka tries to keep leadership there so load stays balanced. Lesson 28.5
preferred leader election (kafka-leader-election.sh, auto.leader.rebalance.enable)
Moving leadership back to each partition's preferred replica: done by the controller's imbalance check when auto.leader.rebalance.enable is true, or by hand with kafka-leader-election. Lesson 28.19
rebalance (PreparingRebalance, consumer rebalance)
The group handing out its partitions again after a member joins, leaves or is kicked out. With the classic protocol nobody consumes during it. Lesson 28.11
record (Kafka record, message)
One entry in a Kafka partition: an optional key, a value, a timestamp and headers, identified by its offset. Lesson 28.1
replica (follower, Kafka replica)
One copy of a partition on one broker. The leader takes reads and writes; followers fetch from it to stay in sync. Lesson 28.5
replication factor (RF, --replication-factor, default.replication.factor)
How many copies of each partition exist, on different brokers. 3 is the usual production value. Lesson 28.5
replication throttle (--throttle, follower.replication.throttled.rate, leader.replication.throttled.rate)
A bandwidth limit for reassignment traffic, per broker, shared by the throttled replicas copying there. Lesson 28.19
retention.bytes (log.retention.bytes)
The size a partition may reach before its oldest closed segments are deleted. Per partition and per replica, -1 (no limit) by default. Lesson 28.15
retention.ms (log.retention.hours, retention)
How long Kafka keeps a closed segment after its newest record (7 days by default); checked every log.retention.check.interval.ms. Lesson 28.15
rolling restart (rolling broker restart)
Restarting brokers one at a time and waiting for under-replicated partitions to reach 0 before the next, so acks=all producers never see NOT_ENOUGH_REPLICAS. Lesson 28.23
Strimzi (KafkaNodePool)
The CNCF Kubernetes operator for Kafka: Kafka, KafkaNodePool, KafkaTopic and KafkaUser resources; it runs the brokers as pods and rolls them one at a time. Lesson 28.23
topic (Kafka topic)
A named stream of records in Kafka, split into partitions that are spread over the brokers. Lesson 28.1
topic override (DYNAMIC_TOPIC_CONFIG)
A config set on one topic (retention.ms, max.message.bytes...); it beats the broker defaults for that topic only and shows in the Configs: field of --describe. Lesson 28.17
unclean leader election (unclean.leader.election.enable)
Letting an out-of-sync replica become leader when no ISR member is available: the partition comes back, acknowledged records may be lost. Off by default. Lesson 28.7
under-replicated partition (URP, UnderReplicatedPartitions)
A partition whose ISR is smaller than its replica list: a broker is down or behind. Expected briefly during restarts and reassignments; a problem when it lasts. Lesson 28.5