A timeout is not a failure. It is an absence of information.

00The core idea

local state   = IN_PROGRESS
remote truth  = { NOT_STARTED | RUNNING | COMPLETED }

IN_PROGRESS is not a final truth.
It is an ownership and recovery state.

Retry and recovery are different jobs. Retry is trying again when the failure is transient or safely repeatable. Recovery is reconstructing what actually happened after a crash, timeout or lost acknowledgement, and then continuing without corrupting anything.

01Timeouts are ambiguous

The three cases, in order of preference

Crash timing windows during an external call

02The decision tree

03Claim, lease, complete

-- claim: handles both concurrent duplicates and later retries
INSERT INTO idempotency_keys (idempotency_key, status, owner_id, lease_expires_at)
VALUES (:key, 'IN_PROGRESS', :worker, NOW() + INTERVAL '5 minutes')
ON CONFLICT DO NOTHING
RETURNING *;
-- row returned → I own execution.  no row → someone else owns or owned it.

-- take over, only from an owner that looks dead
UPDATE idempotency_keys
   SET owner_id = :worker, lease_expires_at = NOW() + INTERVAL '5 minutes'
 WHERE idempotency_key = :key
   AND status = 'IN_PROGRESS'
   AND lease_expires_at < NOW()
RETURNING *;

-- complete, only if I still own it
UPDATE idempotency_keys
   SET status = 'COMPLETED'
 WHERE idempotency_key = :key AND owner_id = :worker;

When the work is replayable, keep it simple

GCS read + in-memory computation are safe to repeat.
Do not add step-level idempotency for them.

  fetch / read / compute        ← replayable, no bookkeeping
  BEGIN
    DB write A
    DB write B
    idempotency = COMPLETED     ← same transaction
  COMMIT

Crash before commit and the transaction rolls back, so the whole thing replays cleanly. Commit succeeds and business state plus completion state are durable together. One transaction is always the simplest correctness model available — reach for anything else only when the work genuinely spans systems.

04Idempotency vs locking vs constraints

The distinction is not “same time versus one after another”. Two duplicate requests can arrive simultaneously and still be handled by idempotency — provided the idempotency claim itself is concurrency-safe, which is what the unique constraint and atomic insert are for.

Request identity is not business identity

a user double-clicks Cancel; the client generates two request IDs

  request-level:   Idempotency-Key = abc123   vs   xyz789   ← different!
  business-level:  (policy_id = P123, operation = CANCEL)   ← the same

so you need both:
  UNIQUE(idempotency_key)               stops retry duplication
  UNIQUE(policy_id, operation_type)     stops duplicate user intent

Two users, one slot — a different problem entirely

-- this is concurrency, not idempotency: two distinct logical operations
UPDATE slots SET status = 'BOOKED'
 WHERE id = :slot_id AND status = 'AVAILABLE';

-- rows_updated = 1 → won.   0 → someone else took it.

A pessimistic alternative is SELECT … FOR UPDATE inside an explicit transaction. Either way, add a database uniqueness constraint as the last line of defence.

05When one key is not enough

{
  "operation_id": "abc123",
  "status": "IN_PROGRESS",
  "steps": {
    "update_db":      "DONE",
    "external_api_a": "DONE",
    "publish_event":  "DONE",
    "external_api_b": "PENDING",
    "audit":          "PENDING"
  }
}

step keys:  abc123:step2 · abc123:step3 · abc123:step4

Writes inside one Postgres database should be one transaction. Once a workflow spans DB, Kafka and external APIs, you are into per-step state, idempotent operations, outbox, and compensation or saga where the business allows it.

06Redis or Postgres for idempotency state?

the Redis + Postgres failure window

  Postgres business update succeeds
    → server crashes
    → Redis COMPLETED key never written
    → retry sees no key
    → duplicate work, unless something else protects it

07Kafka and Cloud Tasks

Both are at-least-once. A message or task can be delivered again if an acknowledgement is lost or the handler fails, so the handler must turn repeated attempts into one logical outcome.

delivery attempt
  → claim / check the operation
  → execute and commit safely
  → mark COMPLETED
  → acknowledge

08End-to-end: a cancellation

user requests cancellation, client supplies an idempotency key
        ↓
backend atomically creates the operation
  key = abc123, policy = P123, status = IN_PROGRESS
        ↓
call the provider
        ↓
   ┌────────────────┬──────────────────────┐
 SUCCESS                          timeout / crash
   │                                      │
mark COMPLETED                  stays IN_PROGRESS
                                          ↓
                                  recovery worker
                                          ↓
                            atomically claim recovery
                                          ↓
                                check provider status
                                  ↙               ↘
                            CANCELLED          ACTIVE / UNKNOWN
                                │                     ↓
                          mark SUCCESS         bounded polling
                                                      ↓
                                            still not cancelled
                                                      ↓
                                          idempotent retry possible?
                                             ↙              ↘
                                           YES               NO
                                    retry same op        reconcile

Multiple recovery workers

-- option 1: skip what someone else holds
SELECT * FROM idempotency_operations
 WHERE status = 'IN_PROGRESS'
   FOR UPDATE SKIP LOCKED;

-- option 2: atomic state transition
UPDATE idempotency_operations SET status = 'RECOVERING'
 WHERE id = :id AND status = 'IN_PROGRESS';
-- 1 row → I own recovery.  0 rows → already claimed.

09Question drill

10Cheat sheet