How to Scrape Multiple Web Pages with Python and Scrapeless

TL;DR:

• Multiple-page scraping needs an input ledger and a bounded queue. Keep every normalized URL's outcome instead of treating a finished loop as proof of coverage.
• URL deduplication and record deduplication solve different problems. A repeated input should not create another request, while repeated content needs a record identity rule.
• An empty page needs positive source evidence. Missing selectors alone do not establish that the source contains no records.
• Export checks should reopen the files. Compare record identifiers in CSV and JSONL, and reconcile completed, empty, and failed pages with the queue.

Scraping several pages changes the job from one extraction into a collection process. The process needs to know what it was asked to visit, which pages it accepted, and where the written records came from.

Learning how to scrape data from multiple web pages therefore starts with scope and bookkeeping. A Python loop is only the acquisition mechanism. This tutorial builds a bounded URL queue, validates each source, preserves page outcomes, deduplicates records, and checks the exported files.

The public demonstration uses Quotes to Scrape through a direct HTTP path. The same program includes a documented Scrapeless Web Unlocker acquisition branch, which requires your account key and remains a labelled prerequisite. Interactive pagination is a separate browser workflow when the source needs it.

Pipeline at a Glance

The pipeline turns an approved URL list into reconciled page outcomes and accepted data files. Preserve those artifacts separately so a failure is not hidden inside an empty export.

Keep the source operation and application schema distinct. The same source can produce several records, no records, or an unusable response, so page counts and record counts require different names.

Choose a URL List or Source Pagination

A stored URL list fits a task whose source scope is already known. Pagination fits a task that discovers the next permitted page from the source's own navigation.

For a list, keep the original inputs and the normalized queue. For pagination, preserve the current page, observed next link or interaction, visited set, and page budget. Do not assume an arbitrary page-number parameter matches the site's actual navigation.

The pagination workflow comparison explains the different advance and stop conditions. This tutorial uses a finite list so every attempted page can be reconciled directly with an input.

Normalize URLs without Erasing Their Meaning

URL normalization should remove differences that are irrelevant to acquisition while retaining parameters that affect the source. In this example, a fragment-only duplicate is removed, while the path and query remain intact.

URI components and reference resolution distinguishes the fragment from the path and query. Avoid deleting query parameters merely because two URLs look similar; they may select different records or filters.

The program also restricts inputs to the approved demonstration host. A real collector should use its source manifest rather than accepting arbitrary newly discovered destinations.

Prerequisites and Installation

The example needs Python with requests and beautifulsoup4 installed in the project environment. The inspected environment used Requests 2.34.2 and Beautiful Soup 4.15.0; record your own project versions when reproducing it.

Direct mode uses the public demonstration pages and needs no Scrapeless API credential. The unlocker mode requires SCRAPELESS_API_KEY, account access to Web Unlocker, and permission to collect the selected source.

Web Unlocker is the response-oriented acquisition product. Its current rendering configuration uses the v2 request endpoint and nested jsRender fields. The API branch asks for HTML, which feeds the same source-specific parser as direct mode.

Review the source scope before execution. The example runs sequentially over a short fixed list; it is not an unbounded crawler or a template for increasing request volume without permission.

Run the Complete Bounded Collector

The collector deduplicates its input URLs, records each queued page's outcome, and exports only accepted records. Its source contract matches the inspected Quotes to Scrape layout.

The demonstration exports author names, tags, record hashes, and source URLs. It uses the quote text internally to construct a content identity but does not reproduce the complete quotations in the exported dataset.

Note: Direct mode was executed against the public demonstration pages. The unlocker branch requires a real Scrapeless API key and account access and was not executed. Its request fields were checked against current documentation; the direct-mode results are not reported as Web Unlocker performance.

import csv
import hashlib
import json
import os
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import urlsplit, urlunsplit
import requests
from bs4 import BeautifulSoup

INPUT = [
    "https://quotes.toscrape.com/page/1/",
    "https://quotes.toscrape.com/page/2/",
    "https://quotes.toscrape.com/page/2/#top",
    "https://quotes.toscrape.com/page/99/",
    "https://quotes.toscrape.com/missing/"
]
MODE = os.environ.get("FETCH_MODE", "direct")
if MODE not in {"direct", "unlocker"}:
    raise ValueError("Unsupported acquisition mode")

def normalize(url):
    parts = urlsplit(url)
    if parts.scheme != "https" or parts.hostname != "quotes.toscrape.com":
        raise ValueError("URL is outside the permitted demo scope")
    return urlunsplit((parts.scheme, parts.netloc.lower(),
                      parts.path or "/", parts.query, ""))

queue = list(dict.fromkeys(normalize(url) for url in INPUT))
pages = []
records = {}
duplicate_records = 0
with requests.Session() as client:
    for url in queue:
        page = {"requested_url": url}
        try:
            if MODE == "unlocker":
                response = client.post(
                    "https://api.scrapeless.com/api/v2/unlocker/request",
                    headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
                    json={"actor": "unlocker.webunlocker",
                          "proxy": {"country": "ANY"},
                          "input": {"url": url, "jsRender": {
                              "enabled": True, "response": {"type": "html"}}}},
                    timeout=60
                )
                response.raise_for_status()
                envelope = response.json()
                html = envelope.get("data")
                if envelope.get("code") != 200 or not isinstance(html, str):
                    raise ValueError("Expected a successful HTML envelope")
                page["final_url"] = None
            else:
                response = client.get(url, timeout=30)
                response.raise_for_status()
                html = response.text
                page["final_url"] = response.url
            soup = BeautifulSoup(html, "html.parser")
            heading = soup.select_one("h1 a")
            grids = soup.select("div.container > div.row > div.col-md-8")
            if not heading or heading.get_text(strip=True) != "Quotes to Scrape" or not grids:
                raise ValueError("Source identity does not match the demo contract")
            grid = grids[-1]
            cards = grid.select("div.quote")
            if not cards and (grid.select_one("ul.pager") is None or
                              "No quotes found!" not in grid.get_text(" ", strip=True)):
                raise ValueError("Empty state is not confirmed")
            extracted = []
            for card in cards:
                text_node = card.select_one("span.text")
                author_node = card.select_one("small.author")
                if not text_node or not author_node:
                    raise ValueError("Required source field missing")
                text = text_node.get_text(strip=True)
                author = author_node.get_text(strip=True)
                if not text or not author:
                    raise ValueError("Required source field empty")
                digest = hashlib.sha256((text + "\n" + author).encode()).hexdigest()
                extracted.append({"record_id": digest, "author": author,
                                  "tags": [x.get_text(strip=True) for x in card.select("a.tag")],
                                  "source_urls": [url]})
            for record in extracted:
                digest = record["record_id"]
                if digest in records:
                    duplicate_records += 1
                    records[digest]["source_urls"].append(url)
                else:
                    records[digest] = record
            page.update(status="completed" if cards else "empty", records=len(cards),
                        captured_at=datetime.now(timezone.utc).isoformat())
        except (requests.RequestException, ValueError, KeyError) as error:
            page.update(status="failed", reason=type(error).__name__)
        pages.append(page)

rows = list(records.values())
with Path("records.jsonl").open("w", encoding="utf-8") as output:
    for row in rows:
        output.write(json.dumps(row, ensure_ascii=False) + "\n")
with Path("records.csv").open("w", encoding="utf-8", newline="") as output:
    writer = csv.DictWriter(output, fieldnames=["record_id", "author", "tags", "source_urls"])
    writer.writeheader()
    for row in rows:
        writer.writerow({**row, "tags": json.dumps(row["tags"], ensure_ascii=False),
                         "source_urls": json.dumps(row["source_urls"])})
with Path("records.csv").open(encoding="utf-8", newline="") as source:
    csv_rows = list(csv.DictReader(source))
jsonl_rows = [json.loads(line) for line in Path("records.jsonl").read_text().splitlines()]
assert [r["record_id"] for r in csv_rows] == [r["record_id"] for r in jsonl_rows]
summary = {"mode": MODE, "input_urls": len(INPUT), "duplicate_inputs": len(INPUT)-len(queue),
           "queued_pages": len(queue), "completed": sum(p["status"]=="completed" for p in pages),
           "empty": sum(p["status"]=="empty" for p in pages),
           "failed": sum(p["status"]=="failed" for p in pages),
           "unique_records": len(rows), "duplicate_records": duplicate_records,
           "csv_rows": len(csv_rows), "jsonl_rows": len(jsonl_rows)}
assert summary["completed"] + summary["empty"] + summary["failed"] == len(queue)
Path("pages.json").write_text(json.dumps(pages, indent=2), encoding="utf-8")
Path("summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
print(json.dumps(summary))

Save the program as multi_page.py and run it in a project directory. With no mode override it uses direct acquisition. Select FETCH_MODE=unlocker only after completing that branch's credentials and source-permission prerequisites.

The program retains one page outcome per unique queued URL. It checks the expected source heading before parsing cards, and it requires the observed no-results message and navigation container before accepting an empty page.

Start Scraping with Scrapeless

Power up your web scraping and automation workflow with Scrapeless!
Sign up today and get $5 in free credit — no credit card required.

Claim your free credit now in the Scrapeless Dashboard.

Extract and Validate Each Record

Record validation should require the fields that identify meaningful source data. The example requires both the quote text used for the hash and the observed author name before committing a page's records.

Selectors belong to the inspected demonstration source: div.quote, span.text, small.author, and a.tag. The main data container is the final matching content column, distinct from the header column. Reinspect these assumptions when the source changes.

The HTTP response semantics describes protocol outcomes, not the correctness of these source fields. A successful response still has to satisfy the application contract.

The parser builds a page's candidate records before adding them to the output set. A missing required field rejects that page instead of silently exporting a partial capture as complete.

Track Completed, Failed, and Empty Pages

Page outcomes should explain what happened to each queued source. The demonstration distinguishes a valid record page, a confirmed empty page, and an HTTP or source-contract failure.

The bounded run used five URL inputs. Removing a fragment-only duplicate left four queued pages. The direct acquisition run produced two completed pages, one confirmed empty page, and one failed page, with 20 unique records written to both files.

These counts describe the bounded public demonstration, not a guaranteed result for another site or a managed acquisition benchmark. The unavailable path is included to test outcome reconciliation; it is not sent repeatedly until it disappears from the failure list.

Deduplicate Records and Preserve Source Lineage

Record deduplication compares a defined content identity after extraction. The example hashes the quote text and author together and preserves source URLs when the same identity appears again.

This is an exact identity rule for the demonstrated record type. A product identifier, revision, or locale may need a different key. Do not use one universal text hash to combine records whose differences matter to the task.

Keep source documentation with accepted data. Deduplication should preserve the relationship to source captures instead of erasing where repeated content was observed.

Export CSV and JSONL, Then Reconcile Them

Export verification should read the artifacts after serialization. The program reopens CSV and JSONL and compares the ordered record identifiers before reporting its summary.

The demonstration produced 20 CSV rows and 20 JSONL records with matching identifiers. summary.json retains those counts, while pages.json preserves the individual page outcomes.

CSV field quoting handles serialization details. Format validity is separate from source acceptance, so both checks are required. The example stores list-valued fields as JSON strings inside CSV cells and as arrays in JSONL.

The conservation checks are straightforward: completed plus empty plus failed pages equals the unique queue size, and exported identifiers match between the files. These checks cannot prove all source facts, but they can expose dropped pages or inconsistent artifacts.

When Pagination Needs Browser Interaction

Interactive pagination needs a browser workflow when the source's next state requires a click or another page action. Rendering one returned HTML document is not automatically a persistent multi-step session.

Choose a dedicated Agent Browser workflow for that case, inspect the actual next control, and wait for the required new records. Keep the same page budget, visited-state tracking, and record acceptance policy around the browser acquisition.

Do not mix the paths invisibly. Record whether the source was obtained through direct HTTP, a Web Unlocker response, or an interactive browser session. A change in acquisition can change page state or content scope.

Handle Source Scope Responsibly

A multi-page task should remain within the approved source list and use only the data required for its purpose. Review terms and the robots exclusion rules before enabling a broader collector.

Keep a conservative per-host budget. This example is sequential; a pilot that uses concurrency should stay within its permitted collection policy, such as a cap of three workers per host rather than an assumed service maximum.

Preserve unresolved outcomes for review. A failure list is part of the completed job's evidence, not an inconvenience to omit from the final report.

Conclusion

Scraping multiple web pages with Python requires a bounded queue, source-specific acceptance, and file reconciliation. The collection process is complete only when each queued input has an outcome and the written artifacts match the accepted records.

Use direct acquisition for suitable permitted static sources, the documented Web Unlocker branch when the task needs its response-oriented path, and a separate browser workflow for interactive pagination. Keep the ledger and output contract stable across those choices.

Ready to Build a Reconciled Collection Job?

Plan a bounded workflow with Scrapeless and evaluate acquisition requirements against the current pricing. Discuss queue and export contracts on Telegram.

FAQ

Q: Does public visibility establish permission to collect multiple pages?

Public visibility does not establish permission for every collection or use. Review source terms and applicable requirements, and consult qualified counsel when the intended activity is uncertain.

Q: When should a collector use a URL list instead of pagination?

Use a URL list when the permitted source set is known. Use source pagination when the task discovers its next page from an inspected link or interaction.

Q: Is an empty selector result a valid empty page?

An empty selector result alone is unresolved. Require the source's positive empty-state evidence and confirm page identity before recording a valid empty outcome.

Q: How can a job show that it handled every input URL?

Preserve the raw inputs, deduplicated queue, and one outcome per queued source. Reconcile completed, empty, and failed outcomes with that queue.

Q: Does the Web Unlocker branch need a separately configured proxy?

The shown managed request uses its documented country field. Separately allocated proxy credentials are needed when your own client uses the standalone proxy product instead.

Q: What should happen to a challenge or access-denied document?

The collector should reject unsuitable source content and retain the acquisition reason. An HTTP success status does not make a challenge document an accepted data page.

Q: What should change when source markup rotates?

Reinspect page identity and the required fields, then update source-specific selectors and empty-state checks. Do not treat changed markup as proof that the source has no records.

Q: How much concurrency should the tutorial use?

The demonstrated collector runs sequentially. A permitted pilot can set a conservative per-host cap such as three workers, then evaluate operation before expanding it.

Q: Does this workflow need an AI agent?

The Python queue, extraction, and export checks run without an AI agent. An agent can consume accepted records or supervise a task around those deterministic components.

Q: Should the requested URL be stored as the final or canonical URL?

Store the requested URL separately from observed final or canonical identity. Direct mode records the response's final URL; the shown API branch leaves that field unknown instead of inventing a target redirect result.