Software Design

Web Crawler URL Frontier State Machine

State Diagrams · made in Skemio

State machine of a web crawler URL frontier: Discovered, Robots Check, Queued, Fetching, Parsed, and Dead Letter states.
State machine of a web crawler URL frontier: Discovered, Robots Check, Queued, Fetching, Parsed, and Dead Letter states.
Open it in SkemioMake your own

What it shows

  • Robots.txt evaluation must happen before network fetching to avoid bans.
  • Parsed targets need deduplication check before queuing to prevent loops.
  • Dead-letter status preserves retry budgets when domains return 4xx/5xx.

The takeaway

An explicit URL state machine guarantees crawl politeness and stops memory bloat.

More state diagrams