HLDArena
HLD-412·Coordination & IDs
Expert · 2,000 pts

Durable Stateful AI Agent Checkpointing & Sandboxed Tool Runner

Design a Temporal / LangGraph-scale durable AI agent orchestrator executing multi-step ReAct loops with exactly-once side-effect tool idempotency and instant crash resumption.

Max p99
≤ 95 ms
Availability
≥ 99.5%
Budget
≤ $11,300

Functional Requirements

  • FR-1POST /agents/{run_id}/step — Atomically checkpoints ReAct state and dispatches idempotent tool call.
  • FR-2GET /agents/{run_id}/state — Streams live reasoning trace and tool outputs to clients.
  • FR-3DELETE /agents/{run_id}/step — Releases canceled or expired leases/holds atomically and immediately refreshes cached availability.
  • FR-4Worker/DB crashes must never execute external mutating tools twice or lose committed checkpoints.

Workload Profile

Steady Autonomous Runs58,000 State Read RPS / 13,000 Checkpoint Write RPS
Enterprise Workflow Trigger Burst155,000 Read RPS / 35,000 Step Write RPS
Active Working Set40 GB active ReAct state graphs & tool idempotency locks
Consistency GateStrict serializable step transitions + RPO = 0 checkpointing

Consistency Model

Strict serializable step-index CAS + durable write-ahead queue so crashed agent runs resume from exact step without duplicate tool side effects.

Design Write-up
0 boxes · 0 edges
0 words
Hidden Cases #4–#7 run automatically on Submit
Case 1: Smoke & API ContractInput Workload
Read RPS
2,000 rps
Write RPS
400 rps
Key Distribution
Uniform (Low Skew)
Chaos Fault
None (Healthy Cluster)
Expected Pass CriteriaASSERTIONS
• Tail Latency: p99 ≤ 190 ms
• Availability: ≥ 99.5%
Valid end-to-end Client → Compute → Storage topology