The Problem
A queue can redeliver a message after a network interruption, consumer timeout, deployment, or retry. If the consumer blindly applies side effects, one candidate update, email, status transition, or batch chunk can run twice.
My Default Pattern
- Give every unit of work a stable idempotency key, usually derived from batch id, chunk id, operation type, and target record.
- Check persisted operation state before doing side effects.
- Write state transitions explicitly:
Pending → Processing → CompletedorFailed. - Make external side effects conditional on the operation not already being completed.
- Acknowledge the message only after state and side effects are consistent enough to recover.
Important distinction:
Idempotency is not the same as exactly-once delivery. I assume messages can repeat, then design the handler so repeated delivery reaches the same final state.
Failure Scenarios
- If the worker crashes before marking complete, the message can retry and resume from persisted state.
- If the worker crashes after a side effect but before final state, the next attempt checks durable evidence before repeating the effect.
- If two workers receive related messages, locks or compare-and-set updates protect critical state transitions.
Trade-off
Idempotency adds schema and state management, but it is cheaper than debugging duplicate side effects in production. For high-volume systems, this is one of the first reliability mechanisms I design.