Design a distributed web crawler that can discover, fetch, parse, deduplicate, and store web content at very large scale. The crawler should start from a known set of seed URLs, continuously fetch pages, extract links, add new URLs back into the crawl frontier, and store fetched content for downstream indexing or analytics.
