HLDArena
HLD-203·Messaging & Queues
Medium · 1,000 pts

Distributed Delayed Job & Cron Scheduler

Design a multi-tenant delayed task scheduler executing millions of cron and future-timestamp jobs with immediate job cancellation visibility.

Max p99
≤ 65 ms
Availability
≥ 99.5%
Budget
≤ $4,400

Functional Requirements

  • FR-1POST /v1/jobs — Schedules or cancels a job in queue_id for a target execution timestamp.
  • FR-2GET /v1/queues/{queue_id}/due — Polls ready-to-run tasks and status for worker pools.
  • FR-3PATCH & DELETE /v1/jobs/{queue_id} — Updates, cancels, or deletes queue_id records and immediately invalidates or updates cached state (read-after-write).
  • FR-4Canceled jobs must immediately reflect as canceled (read-after-write) so workers do not fire stale schedules.

Workload Profile

Baseline Load20,000 Poll Read RPS / 9,000 Schedule Write RPS
Minute-Boundary Spike50,000 Read RPS / 24,000 Write RPS
Active Working Set16 GB upcoming 15-minute execution bucket
Traffic DistributionSharp synchronized burst at HH:MM:00 boundaries

Consistency Model

Read-after-write consistency required for job edits/cancellations; zero lost scheduled jobs on primary DB crash.

Design Write-up
0 boxes · 0 edges
0 words
Hidden Cases #4–#7 run automatically on Submit
Case 1: Smoke & API ContractInput Workload
Read RPS
2,000 rps
Write RPS
400 rps
Key Distribution
Uniform (Low Skew)
Chaos Fault
None (Healthy Cluster)
Expected Pass CriteriaASSERTIONS
• Tail Latency: p99 ≤ 130 ms
• Availability: ≥ 99.5%
Valid end-to-end Client → Compute → Storage topology