System Design

Web Crawler Pipeline: Billion-Page Scale

System Design diagram · made in Skemio

System architecture diagram of a web crawler showing the URL frontier, DNS resolver, fetcher workers, Bloom filter deduplicator, and parser pipeline.
System architecture diagram of a web crawler showing the URL frontier, DNS resolver, fetcher workers, Bloom filter deduplicator, and parser pipeline.
Open it in SkemioMake your own

What it shows

  • URL Frontiers use dual queues to balance strict politeness with crawling priority.
  • Bloom filters detect duplicate URLs in-memory with near-zero memory footprint.
  • Distributed parsers strip HTML, extract outgoing links, and feed the frontier.

The takeaway

Politeness-controlled queues and in-memory Bloom filters prevent crawler bans and redundant work.

More system design diagrams