One viral post, one very unhappy shard.

00The core idea

post:101:likes  →  one key  →  one hash slot  →  one primary  →  HOT SHARD

no amount of adding primaries changes that chain

01Logical bucketing

Split one logical counter into many independently hashable keys, and route each user deterministically: bucket = stable_hash(user_id) % N. The single-key bottleneck disappears because there is no longer a single key.

The cost is that a read now needs an aggregation step — either summing buckets at read time, or maintaining a periodically recomputed total so the read path stays one GET.

02Two hashing layers, two jobs

application routing
  user_id  →  stable_hash(user_id) % N  →  logical bucket

Redis routing
  post_id + bucket  →  CRC16(key) % 16384  →  hash slot  →  physical primary

They are not alternatives and neither substitutes for the other. Redis will happily balance slots perfectly and still leave you with one scorching key, because balancing slots is not the same as balancing load.

03Redis Cluster, slots and resharding

A cluster has exactly 16,384 hash slots, assigned to primaries — not necessarily contiguously. A shard is a primary plus replicas; replicas give availability and optional read scaling, and add no primary write capacity.

The client is cluster-aware: it discovers topology, caches the slot-to-primary map, computes the slot for each key and talks to the right primary directly. It is not a coordinator — every app process has its own. It also handles ASK (this key has moved for now, during migration) and MOVED (slot ownership changed permanently — refresh your map).

resharding 10 primaries → 11

  before: 10 primaries ≈ 1,638 slots each
  after:  11 primaries ≈ 1,489 slots each

a balanced rebalance takes slots from several primaries, not one.
application keys never change — only which primary owns their slots.

04How many buckets?

Logical bucket count N and physical shard count M are independent. With 64 buckets and 16 shards the average is 4 buckets per shard — but the actual placement is probabilistic. A shard may get 7, or 2, or none for that particular post. Redis balances slots, not “buckets per post”.

skew math, for interviews

  buckets landing on one shard ≈ Binomial(N, 1/M)

  mean   μ = N / M
  stddev σ = sqrt(N · (1/M) · (1 − 1/M))

  typical hottest shard ≈ μ + sqrt(2 ln M) · σ
  for M ≈ 10, sqrt(2 ln M) ≈ 2.15

then convert back to QPS:  per-bucket QPS = total QPS / N
never capacity-plan from total QPS / shard count alone

Use substantially more logical buckets than physical shards, so placement luck stops mattering — but not absurdly more, because every bucket adds keys, metadata, aggregation work and operational surface.

05Three knobs: N, M and B

This is the heart of the topic. At 10M QPS with only 64 buckets, one bucket carries ~156K QPS — already past what a primary comfortably serves. Scaling from 10 to 100 primaries changes nothing for that key, because one key still maps to one slot on one primary. N has to rise first.

And the corollary worth stating: you can scale 1M → 10M without batching at all, given enough buckets and enough primaries. Batching is a cost and efficiency optimisation, not the only survival route.

06Kafka batching, done safely

Ten million like events per second need not become ten million Redis commands. Consumers can aggregate increments for the same (post_id, bucket) and issue one INCRBY key 10000 instead of ten thousand INCRs.

atomic, in one Lua script:

  if incoming_offset already applied:
      return DUPLICATE
  INCRBY counter delta
  update last_processed_offset

then, and only then, commit the Kafka offset

A high-watermark offset is only safe if that partition is processed sequentially — otherwise a later offset can complete first and make an earlier one look like a duplicate.

07Changing N safely

N can be raised dynamically from per-post metrics — write QPS, queue lag, latency contribution. The catch is that changing N changes the modulo mapping:

old: bucket = stable_hash(user_id) % 64
new: bucket = stable_hash(user_id) % 256

the same user now maps somewhere else — so the cutover needs
explicit semantics, usually versioned routing:

  post:101 metadata
    mode         = SHARDED
    version      = 3
    bucket_count = 256

  new writes → post:101:v3:likes:[bucket]
  total      = sum(v1) + sum(v2) + sum(v3)

Promotion works the same way: a normal post is one key, and when it crosses a threshold the existing count is frozen as a base while new likes go to buckets — total = base + sum(buckets). At very large scale it is often simpler to use stable bucketing from the start than to operate a promotion path.

08What is automatic, what you decide

09Question drill

10Cheat sheet