Problem
Long-running bulk actions should not block API requests or depend on one process staying alive. The system needs to accept work quickly, split it into smaller operations, run those operations in parallel, and expose accurate progress.
Professional Case Study · Distributed Systems
Distributed Task Processing Platform
A company-owned asynchronous processing platform built with ASP.NET Core, RabbitMQ, MassTransit, Redis, MySQL, and Docker. My work covered chunked workloads, parallel consumers, idempotency, retries, dead-letter handling, distributed locks, and status tracking.
Confidentiality: Source code and internal implementation details are private. This case study describes my contribution and general engineering decisions without disclosing proprietary information.
Client submits bulk task
↓
ASP.NET Core API validates request and creates batch state
↓
RabbitMQ publishes chunk messages through MassTransit
↓
Workers process chunks with Redis locks and idempotency checks
↓
MySQL stores task, chunk, status, retry, and completion state
Key Decisions
- Separate request acceptance from execution so the API returns a tracking handle quickly.
- Represent bulk work as batches and chunks so throughput scales by adding consumers.
- Use idempotency keys per chunk to make replays and retries safe.
- Use Redis locks for short critical sections where duplicate workers could race.
- Persist status in MySQL so progress survives worker restarts.
Failure Handling
- Transient failures retry with bounded attempts and delay.
- Poison messages move to a dead-letter path for inspection instead of blocking the queue.
- Stale in-progress chunks can be requeued after timeout checks.
- Completion is derived from persisted chunk state, not from a worker's memory.
Trade-offs
- Chunking adds bookkeeping, but keeps large workloads recoverable and visible.
- At-least-once delivery is simpler and more reliable than trying to promise exactly-once processing.
- Redis locks reduce race conditions, but every lock must have ownership and expiry rules.
Results
The system makes backend reliability choices visible: how messages move, how workers recover, how duplicate processing is avoided, and how a user can check task status without needing to understand the queue internals.