Python Web Crawler Tutorial: Requests to Playwright

TL;DR:

• A Python web crawler is a queue with rules. It starts from one or more URLs, normalizes discovered links, rejects duplicates, enforces scope, and records what happened to every page.
• Requests and Beautiful Soup are the right baseline for server-rendered pages. They keep the crawl fast and make failures easy to inspect.
• Playwright belongs on the JavaScript branch, not on every URL. Render only pages whose required content is absent from the initial response.
• Scrapeless Scraping Browser moves browser execution out of the crawler process. It is useful when managed sessions and dynamic rendering become the main operational cost.

---

A crawler decides where to go next from what it just fetched. That makes URL policy, queue state, and deduplication more important than any single CSS selector.

This tutorial builds the design in layers: plain HTTP for stable pages, browser rendering for client-side content, then managed browser execution when local Chromium becomes the constraint.

What Is a Python Web Crawler?

A Python web crawler is a program that discovers pages by following links from a seed set. Web scraping extracts fields from a known page; crawling discovers which pages exist and schedules them for later processing.

The two jobs often share one process, but they should remain separate in the design. Discovery returns canonical URLs and relationship metadata. Extraction returns records such as title, price, date, or body text.

The Crawler Pipeline at a Glance

Stage	Input	Output	Main rule
Seed	Starting URLs	Initial queue	Use explicit allowed domains
Fetch	Canonical URL	HTTP response or rendered page	Set a clear timeout
Discover	HTML document	Candidate links	Resolve relative URLs
Normalize	Candidate URL	Canonical URL	Remove fragments and normalize host casing
Filter	Canonical URL	Accepted or rejected URL	Enforce domain, path, and depth scope
Deduplicate	Accepted URL	New queue item or existing record	Key on the canonical form
Extract	Page content	Structured record	Treat optional fields as nullable
Store	Record plus provenance	Durable dataset	Preserve source URL and collection time

Path A: Requests and Beautiful Soup for Static Pages

Requests is appropriate when the initial response already contains the links and fields the crawler needs. Beautiful Soup then turns the HTML into a searchable tree.

Start with a deque for breadth-first traversal, a set for visited canonical URLs, and a hard page limit. Resolve relative links with urljoin, then check the hostname after resolution. The URI generic syntax defines how relative references and fragments fit the URL model.

Do not mark a URL as visited only after a successful fetch. Mark it when it enters the queue; otherwise two parent pages can schedule the same child before either request completes.

Normalize URLs Before Deduplication

URL normalization determines whether /products, /products#reviews, and an equivalent absolute URL become one crawl target or three. A conservative canonicalizer should:

• lowercase the scheme and hostname;
• remove fragments;
• resolve . and .. path segments;
• remove default ports;
• preserve query parameters unless their meaning is known;
• reject non-HTTP schemes before scheduling.

The WHATWG URL Standard documents the parser behavior implemented by modern browsers. Keep normalization conservative because removing an unknown query parameter can collapse distinct resources.

Enforce Scope, Depth, and Crawl Budget

A crawler needs explicit stopping rules before it sends the first request. Define allowed hosts, accepted path prefixes, maximum depth, and maximum pages. Depth is the number of link edges from the seed, not a proxy for site hierarchy.

Store the discovery parent with each queue item. That creates a URL graph that helps explain why a page was visited and makes infinite calendar or faceted-navigation patterns visible.

Start Scraping with Scrapeless

Power up your web scraping and automation workflow with Scrapeless!
Sign up today and get $5 in free credit — no credit card required.

Claim your free credit now in the Scrapeless Dashboard.

Respect Robots and Site Policy

Robots directives are part of crawl planning, not an afterthought. The Robots Exclusion Protocol defines how crawlers discover and interpret robots.txt rules.

Check applicable terms and law, identify the crawler where appropriate, and keep collection limited to permitted public pages. A robots rule is not an authorization mechanism, so an allowed path does not replace legal and contractual review.

Path B: Playwright for JavaScript Pages

Playwright is appropriate when the required links or fields appear only after client-side execution. Route those pages to a browser branch after proving the initial response is incomplete.

Use domcontentloaded as the starting navigation event, then wait for a specific page state tied to the data. The Playwright locator guidance recommends role, label, text, and other user-facing attributes when they match the target.

Close each page and browser context after extraction. Browser resources should not live in the same unbounded loop as lightweight HTTP requests.

When to Move Browser Execution to Scrapeless

Scrapeless Scraping Browser is useful when the crawler’s browser branch needs managed sessions, geographic routing, and production browser capacity. The Scraping Browser quickstart covers the current connection path.

Keep the crawler’s queue, URL rules, extraction schema, and storage under application control. Move only the browser execution layer. This separation lets static pages stay on Requests while dynamic pages use a managed browser.

Store Crawl State for Resumption

Persist queued, in-progress, completed, rejected, and failed states separately. Store the canonical URL, discovery parent, depth, fetch method, response status, content hash, and parser version.

A content hash prevents unchanged pages from entering downstream processing again. A parser version makes schema changes traceable when a selector update alters historical output.

Review Scrapeless pricing after measuring the share of pages that actually require browser execution. The Scraping Browser best-practices guide explains how to operate browser-backed work reliably once pages cross that boundary.

Conclusion

A dependable Python crawler is a controlled URL graph. Start with Requests and Beautiful Soup, normalize before deduplication, enforce scope before scheduling, and route only JavaScript-dependent pages through Playwright or Scrapeless Scraping Browser.

Ready to Build a Controlled Crawl Pipeline?

Join developers building web-data pipelines on Discord or Telegram. Create an account at app.scrapeless.com and measure the browser branch against your real URL set.

FAQ

Q: What is the difference between crawling and scraping?

Crawling discovers and schedules pages; scraping extracts fields from a page. One pipeline can do both, but separate states and outputs make it easier to operate.

Q: Is web crawling legal?

Web crawling can be lawful when it accesses permitted public pages, but requirements vary by jurisdiction and site. Review applicable terms, privacy obligations, and legal advice for the use case.

Q: Does a Python crawler need a proxy?

A proxy is useful when authorized collection requires geographic routing or distributed egress. It does not change the crawler’s obligation to respect scope and policy.

Q: How should a crawler handle an Access Denied page?

Record the page as a distinct access outcome, stop extraction for that URL, and review the session, egress, and authorization assumptions before continuing the workflow.

Q: What happens when the DOM changes?

Re-check the discovery and extraction selectors against saved fixtures, update the parser version, and treat absent fields as nullable until the new schema is confirmed.

Q: How much concurrency should a crawler use?

Start with no more than three workers per host and lower the limit when site policy or server behavior requires it. Browser jobs should use a separate, tighter capacity pool.

Q: Can the crawler run without an AI agent?

Yes. The queue, URL policy, HTTP client, browser branch, and storage pipeline can run as ordinary Python services without a model.