Partitions, rebalancing and the hot key that won't split.

00The 10-second model

01Partitions and ordering

A partition is an append-only log. Offsets are partition-local and monotonically increasing. Records are retained by policy even after consumption, which is what makes replay possible — consuming does not delete anything.

Choosing the key

The question that settles it: what is the smallest unit for which the business genuinely requires ordering? Partition by that, and no finer.

02Consumer groups and the parallelism ceiling

Partition count is therefore a capacity-planning input, not an implementation detail. If you will need 40-way consumer parallelism later, a 4-partition topic cannot provide it without repartitioning — and over-partitioning has its own metadata and coordination costs, so it is a real trade-off rather than “pick a big number”.

Why one owner per group at all? Because concurrent consumers of the same partition would complete records out of order, and Kafka's model exists to make ordered consumption tractable. You can parallelise inside a consumer — but then ordering and offset bookkeeping become your problem, not Kafka's.

03Rebalancing

A group's membership changes; every subscribed partition still needs exactly one owner inside that group. Rebalancing is the protocol that converges from the old ownership map to a valid new one, coordinated by a broker acting as group coordinator (not the cluster controller — different job).

Eager vs cooperative

Classic eager rebalancing revokes broad ownership before redistributing — processing pauses and lag spikes, which hurts most during frequent deploys. Sticky and cooperative approaches keep as much of the previous assignment as possible and move ownership incrementally: if C3 joins a group where C1 owns P0–P1 and C2 owns P2–P3, only P3 needs to move.

Modern Kafka also offers a newer server-driven consumer protocol (group.protocol=consumer). Treat it as a version detail after explaining the timeless idea: membership changes force ownership changes. Static membership can reduce churn from short restarts, but it is an optimisation, not a substitute for handling duplicates.

04Offsets and the failure window

read M100
   ↓
external DB write succeeds
   ↓
CRASH
   ✗ offset commit never happens
   ↓
restart / rebalance
   ↓
M100 is delivered again

This is the whole reason “at-least-once” is the default mental model. The side effect and the offset commit are not one atomic transaction with an arbitrary external database, so one of them lands first and a crash between them is always possible.

The practical fix is a unique event_id with a uniqueness constraint, treating duplicate-key as “already processed”, or a sink-specific transactional pattern where one exists.

05Hot partitions

The cluster may have enormous aggregate capacity while being completely blocked by P2's leader broker and P2's single consumer owner. If that consumer processes 20K/s and the partition receives 100K/s, lag grows at 80K/s no matter how many idle consumers are standing by.

Replication does not help either, and it is a common confusion: replicas are copies for durability and failover. There is one leader accepting writes per partition. Replication factor 3 does not mean three consumers can share the work.

06Splitting the hot key — and what it costs

The producer derives a shard from user_id, comment_id or another well-distributed attribute, producing multiple distinct Kafka keys so one hot entity can occupy many partitions.

Static vs dynamic sharding

Sharding every key costs complexity for keys that do not need it. Dynamic sharding — a control-plane record saying this entity currently uses 32 shards — avoids that, and immediately creates a distributed configuration problem instead:

• When exactly do producers switch from 1 shard to 32?
• Can events in the old and new scheme coexist in flight?
• Do consumers need a partitioning_version on every event?
• What does “ordering” even mean during the transition?
• How do you safely shrink the shard count later?

That list is why hot-key splitting is genuinely staff-level: the local fix is easy, and it exports a versioning and migration problem to everyone downstream.

07Detecting skew

the topic average lies

  P0    1,000/s
  P1    1,000/s
  P2  101,000/s   ← the actual problem
  P3    2,000/s
  ───────────────
  average ≈ 26,000/s   ← tells you nothing

08Question drill

09What not to say

10Cheat sheet