HLD-408·Caching & Consistency
Expert · 2,000 ptsMulti-Cluster LLM Prefix KV-Cache Router & Token Scheduler
Design an OpenAI / vLLM multi-cluster inference control plane that routes shared system-prompt prefixes to sticky KV-cache pools, enforces TPM quotas, and survives 8x storage/checkpoint brownouts.
Max p99
≤ 65 ms
Availability
≥ 99.6%
Budget
≤ $11,600
Functional Requirements
- FR-1POST /v1/infer/schedule — Matches prompt prefix_hash to warm KV-cache shard and deducts tenant TPM budget.
- FR-2GET /v1/infer/kv_directory/{prefix_hash} — Looks up live prefix residency and cluster queue depth.
- FR-3PATCH & DELETE /v1/infer/schedule/{prefix_hash} — Updates, cancels, or deletes prefix_hash records and immediately invalidates or updates cached state (read-after-write).
- FR-4Must maintain read-after-write KV residency & TPM accounting while isolating 8x gray latency brownouts.
Workload Profile
Steady Production API78,000 Prefix Route Read RPS / 12,000 Schedule Write RPS
Global Agentic Burst205,000 Read RPS / 30,000 Schedule Write RPS
Active Working Set58 GB distributed prefix KV-directory & tenant TPM token buckets
Fault Scenario8x storage/checkpoint brownout in Wave 3
Consistency Model
Read-after-write on prefix eviction/registration and tenant TPM token ledger.
Solutions Locked
Sign in and either achieve AC or complete 3 official submissions on this question to inspect community diagrams and writeups.
Loading solutions...
Submission History (0)Select to replay
No submissions yet
Click Submit in the top bar to evaluate against all 7 judge cases and save each run here.
Design Write-up
0 boxes · 0 edges
0 words