How I Classify Failures
- Transient: temporary network, database, or dependency issues. Retry with delay.
- Recoverable business state: prerequisite state is not ready yet. Requeue or delay until the state can exist.
- Poison: invalid payload, unsupported state, or repeated failure. Move to dead-letter handling.
Retry
Retry is for errors likely to succeed later without changing the message. I use bounded attempts, delay, and logging that preserves the operation id and attempt count.
Requeue
Requeue is for work that is valid but premature. For example, a chunk may depend on batch state that has not been committed yet. Requeue should be deliberate, not an infinite loop.
Dead-Letter Queue
A DLQ is not a trash can. It is an inspection lane. Messages in the DLQ should include enough context to diagnose the failure and decide whether to repair, replay, or discard.
The queue should keep moving. One bad message should not block unrelated work, and one flaky dependency should not create uncontrolled retry storms.
Trade-off
More explicit failure handling means more code paths, but it makes production behavior easier to reason about. In high-volume systems, predictable failure behavior matters as much as the happy path.