Design a Web Crawler

Functional Requirements:

Non-Functional Requirements:

High Level Design:

Scaling the design: