HLDArena
HLD-229·Geo & Telemetry
Medium · 1,000 pts

Distributed Tracing Span Collector & Tail Sampler

Design a Dapper / Jaeger / Datadog APM tracing backend that reassembles microservice spans by trace_id for tail-based error sampling and waterfall visualization.

Max p99
≤ 75 ms
Availability
≥ 99.4%
Budget
≤ $5,650

Functional Requirements

  • FR-1POST /v1/traces/spans — Ingests high-volume RPC spans routed consistently by trace_id.
  • FR-2GET /v1/traces/{trace_id} — Fetches assembled end-to-end RPC waterfall graph.
  • FR-3PATCH & DELETE /v1/traces/spans/{trace_id} — Updates, revokes, or deletes trace_id entries with bounded TTL or explicit cache/index invalidation.
  • FR-4Must isolate 6x trace storage latency spikes without blocking instrumented production services.

Workload Profile

Steady RPC Spans12,000 Waterfall Read RPS / 18,000 Span Batch Write RPS
Latency Incident Spike32,000 Read RPS / 44,000 Span Write RPS
Active Working Set24 GB in-flight tail-sampling trace buffers
Traffic DistributionWrite-dominant span stream keyed by trace_id

Consistency Model

Eventual trace assembly (< 5s span flush window); ingestion path must never backpressure callers.

Design Write-up
0 boxes · 0 edges
0 words
Hidden Cases #4–#7 run automatically on Submit
Case 1: Smoke & API ContractInput Workload
Read RPS
1,200 rps
Write RPS
400 rps
Key Distribution
Uniform (Low Skew)
Chaos Fault
None (Healthy Cluster)
Expected Pass CriteriaASSERTIONS
• Tail Latency: p99 ≤ 150 ms
• Availability: ≥ 99.4%
Valid end-to-end Client → Compute → Storage topology