HLD-209·Search & Indexing
Medium · 1,000 ptsDistributed Web Crawler URL Frontier & Dedup
Design a URL discovery, deduplication, and politeness-throttled crawl frontier coordinating thousands of concurrent fetcher workers.
Max p99
≤ 55 ms
Availability
≥ 99.5%
Budget
≤ $4,500
Functional Requirements
- FR-1POST /frontier/discover — Checks extracted outbound URLs against seen-set and enqueues unseen links.
- FR-2GET /frontier/{domain_hash}/next — Leases next polite URL batch for a target host.
- FR-3PATCH & DELETE /frontier/discover/{domain_hash} — Updates, revokes, or deletes domain_hash entries with bounded TTL or explicit cache/index invalidation.
- FR-4Must sustain throughput when deduplication storage latency spikes 6x.
Workload Profile
Baseline Crawl26,000 Lease Read RPS / 11,000 Discovery Write RPS
Sitemap Flood Peak70,000 Read RPS / 28,000 Write RPS
Active Working Set32 GB Bloom/fingerprint cache & host queues
Traffic DistributionSkewed link discovery toward large domain_hash hosts
Consistency Model
Eventual deduplication acceptable (rare duplicate fetch tolerated; per-domain politeness leases durable).
Solutions Locked
Sign in and either achieve AC or complete 3 official submissions on this question to inspect community diagrams and writeups.
Loading solutions...
Submission History (0)Select to replay
No submissions yet
Click Submit in the top bar to evaluate against all 7 judge cases and save each run here.
Design Write-up
0 boxes · 0 edges
0 words