HLDArena
HLD-316·Storage & Media
Hard · 1,500 pts

S3-Scale Multipart Blob Metadata & Placement Index

Design AWS S3 / GCS’s strongly consistent bucket-key metadata index, multipart upload commit coordinator, and erasure-coded extent locator.

Max p99
≤ 58 ms
Availability
≥ 99.5%
Budget
≤ $8,600

Functional Requirements

  • FR-1PUT /buckets/{bucket_key}/complete — Atomically commits multipart extent manifest with strong read-after-write.
  • FR-2GET /buckets/{bucket_key} — Resolves extent storage nodes and streams object bytes.
  • FR-3PATCH & DELETE /buckets/{bucket_key}/complete — Updates, cancels, or deletes bucket_key records and immediately invalidates or updates cached state (read-after-write).
  • FR-4Must provide strong read-after-write object listing/HEAD consistency and survive metadata shard crashes.

Workload Profile

Steady Cloud Workloads52,000 GET/HEAD RPS / 8,500 PUT Commit RPS
Spark/ML Checkpoint Surge138,000 Read RPS / 24,000 PUT Write RPS
Active Working Set36 GB hot bucket keyspace prefixes & extent maps
Consistency GateStrong read-after-write on PUT/DELETE (modern S3 semantics)

Consistency Model

Strong read-after-write consistency on object PUT/DELETE metadata across partitioned key ranges.

Design Write-up
0 boxes · 0 edges
0 words
Hidden Cases #4–#7 run automatically on Submit
Case 1: Smoke & API ContractInput Workload
Read RPS
2,000 rps
Write RPS
400 rps
Key Distribution
Uniform (Low Skew)
Chaos Fault
None (Healthy Cluster)
Expected Pass CriteriaASSERTIONS
• Tail Latency: p99 ≤ 116 ms
• Availability: ≥ 99.5%
Valid end-to-end Client → Compute → Storage topology