Crawlee for Python: Queue, Dedupe, and Render a Real Crawl

TL;DR:

• Crawlee for Python gives you a request queue, automatic URL deduplication, enqueue_links(), and a dataset writer, so a paginated crawler is one handler function.
• Crawlee's storage is shared per process, not per crawler. purge_on_start is True and still does not isolate two crawlers in one script.
• Measured: two identical crawlers with max_requests_per_crawl=2 in one script. The first finished 2 requests and wrote 20 items; the second finished 3 requests and reported 50 items across 5 pages.
• BeautifulSoupCrawler does not run JavaScript. On a client-rendered page it finished the request and produced 0 items.
• Crawlee's HttpClient base class is four methods. Implementing one that calls the Scrapeless Universal Scraping API returned 10 items from that same page, with the router handler unchanged.
• The Scrapeless free plan covers every request in this guide.

Crawlee for Python is the part of a scraper you would otherwise write yourself: the queue that holds URLs, the set that stops you fetching one twice, the concurrency limiter, and the writer that puts results on disk. You supply a handler that receives a parsed page.

Because Crawlee owns the queue and the storage, its defaults decide what your results look like — and two of them produce numbers that are wrong in ways no exception will tell you about.

This guide builds a working crawler against a live site, measures what the storage default does to a second crawler, then swaps the transport so the same handler works on a page that renders in the browser.

What Crawlee Gives You

Crawlee ships several crawler classes that share one interface. The one you pick decides how the page is parsed:

• BeautifulSoupCrawler and ParselCrawler fetch over HTTP and hand your handler a parsed tree.
• HttpCrawler gives you the raw response with no parsing.
• PlaywrightCrawler and AdaptivePlaywrightCrawler drive a real browser.

All of them accept the same router, the same concurrency settings, and the same storage. Swapping between them changes the context object your handler receives, which is why moving from an HTTP crawler to a browser crawler is not a one-line change.

Install

pip install 'crawlee[beautifulsoup]'

The extra matters — the base crawlee package does not pull in Beautiful Soup. The verification run used crawlee 1.9.0 with beautifulsoup4 4.15.0 on Python 3.12.

Your First Crawler

A crawler is a class plus a decorated handler. The handler receives a context carrying the parsed page, the request, and the methods for pushing data and queueing more URLs.

def build(*, storage_dir: str | None = None, http_client=None, max_requests: int = 3):
    crawler = BeautifulSoupCrawler(
        http_client=http_client,
        max_requests_per_crawl=max_requests,
        concurrency_settings=ConcurrencySettings(desired_concurrency=2, max_concurrency=2),
        configuration=Configuration(storage_dir=storage_dir) if storage_dir else None,
    )

    @crawler.router.default_handler
    async def handler(context: BeautifulSoupCrawlingContext) -> None:
        for quote in context.soup.select("div.quote"):
            await context.push_data({
                "text": quote.select_one("span.text").get_text(strip=True),
                "author": quote.select_one("small.author").get_text(strip=True),
                "url": context.request.url,
            })
        await context.enqueue_links(selector="li.next a")

    return crawler

context.soup is a Beautiful Soup object, so existing selectors carry over unchanged. context.push_data() appends to the dataset. context.enqueue_links(selector=...) finds anchors matching that selector, resolves each one against the current page using the base-URL rules in the WHATWG URL Standard, and adds the results to the queue — already deduplicated, so a "next" link that points back at a visited page costs nothing.

ConcurrencySettings rejects a max_concurrency below its desired_concurrency, so set both when you lower it.

Run against three pages of a live quotes site:

  static
    requests finished : 3
    dataset items     : 30
    distinct pages    : 3
    first quote       : “The world as we have created it is a process of our think
    first author      : Albert Einstein

Three requests, ten quotes each, three distinct source URLs. The handler never built a URL or tracked a visited set.

Where the Data Goes

push_data() writes to a dataset under ./storage, and crawler.get_data() reads it back:

async def report(label, crawler, start_url):
    await crawler.run([start_url])
    data = await crawler.get_data()
    print(f"  {label}")
    print(f"    requests finished : {crawler.statistics.state.requests_finished}")
    print(f"    dataset items     : {data.count}")
    print(f"    distinct pages    : {len({i['url'] for i in data.items})}")
    return data

crawler.statistics.state.requests_finished is the count Crawlee actually completed, which is worth printing next to the dataset count. When those two disagree with what you expect, the reason is usually the next section.

The Storage Outlives Your Crawler

Configuration().purge_on_start is True. That reads like a guarantee that every run starts from an empty dataset and an empty queue. It is not — the purge happens once, when storage is first opened in the process, so a second crawler built in the same script joins the storage the first one left behind.

Two crawlers, built by the same function, both limited to two requests, both starting from the same URL:

await report("crawler A", build(max_requests=2), "https://quotes.toscrape.com/")
await report("crawler B", build(max_requests=2), "https://quotes.toscrape.com/")

  crawler A
    requests finished : 2
    dataset items     : 20
    distinct pages    : 2
  crawler B
    requests finished : 3
    dataset items     : 50
    distinct pages    : 5

Crawler B was configured for two requests and finished three. Its dataset reports 50 items across 5 distinct pages, which includes everything crawler A wrote. Nothing raised, and both runs logged as successful.

The start URL crawler B was given had already been visited, so deduplication discarded it, while the page crawler A had queued and never reached was still waiting. The limit and the dataset a crawler reports are both properties of the shared storage, not of that crawler.

Give each crawler its own storage directory when they share a process. That is the one argument the builder above takes for exactly this reason:

        configuration=Configuration(storage_dir=storage_dir) if storage_dir else None,

Re-run the same three stages with storage_dir set per crawler and the counts become the ones you configured. One crawler per process is the other answer, and the simpler one for production.

When the Page Renders in the Browser

https://quotes.toscrape.com/js/ builds its DOM from a JavaScript array. BeautifulSoupCrawler fetches it without complaint:

  javascript
    requests finished : 1
    dataset items     : 0
    distinct pages    : 0

One request finished, zero items. The markup Crawlee received contains no div.quote elements, and an HTTP crawler has nothing that would create them.

The documented answer is PlaywrightCrawler, which means a browser dependency, a different context object, and a rewrite of the handler's parsing. The narrower change is to keep BeautifulSoupCrawler and replace only its transport, which Crawlee supports through the http_client parameter.

HttpClient is four methods, and only two of them need real work. A response object satisfying Crawlee's HttpResponse structural type — a protocol in the sense of the Python typing protocol specification — wraps the rendered HTML. It has to expose a status code and headers because Crawlee treats them the way the HTTP semantics specification defines them:

class RenderedResponse:
    """Adapts a rendered HTML string to Crawlee's HttpResponse protocol."""

    def __init__(self, body: bytes, status_code: int = 200) -> None:
        self._body = body
        self._status_code = status_code

    @property
    def http_version(self) -> str:
        return "HTTP/1.1"

    @property
    def status_code(self) -> int:
        return self._status_code

    @property
    def headers(self) -> HttpHeaders:
        return HttpHeaders({"content-type": "text/html; charset=utf-8"})

    async def read(self) -> bytes:
        return self._body

    async def read_stream(self) -> AsyncIterator[bytes]:
        raise RuntimeError("streaming is not supported by this client")
        yield b""

The client itself calls the Scrapeless Universal Scraping API. That endpoint renders the page and returns the HTML as a string. The blocking call goes through asyncio.to_thread so it does not stall the event loop that the asyncio task documentation describes:

class ScrapelessHttpClient(HttpClient):
    """Routes every Crawlee request through the Universal Scraping API."""

    def __init__(self, *, proxy_country: str = "US") -> None:
        super().__init__()
        self._proxy_country = proxy_country
        self._token = os.environ["SCRAPELESS_API_KEY"]

    def _render(self, url: str) -> bytes:
        payload = {
            "actor": "unlocker.webunlocker",
            "input": {"url": url, "proxy_country": self._proxy_country, "js_render": True},
        }
        request = urllib.request.Request(
            UNLOCKER,
            data=json.dumps(payload).encode(),
            headers={"Content-Type": "application/json", "x-api-token": self._token},
        )
        with urllib.request.urlopen(request, timeout=180) as response:
            return json.loads(response.read().decode())["data"].encode("utf-8")

    async def crawl(self, request, *, session=None, proxy_info=None, statistics=None,
                    timeout: timedelta | None = None) -> HttpCrawlingResult:
        body = await asyncio.to_thread(self._render, request.url)
        return HttpCrawlingResult(http_response=RenderedResponse(body))

    async def send_request(self, url, *, method="GET", headers=None, payload=None,
                           session=None, proxy_info=None, timeout=None) -> HttpResponse:
        body = await asyncio.to_thread(self._render, url)
        return RenderedResponse(body)

    def stream(self, url, **kwargs):
        raise NotImplementedError("this client does not stream")

    async def cleanup(self) -> None:
        return None

Pass it to the same crawler and run the same page:

  javascript+api
    requests finished : 1
    dataset items     : 10
    distinct pages    : 1
    first quote       : “The world as we have created it is a process of our think

Ten items from the page that produced zero. The router handler, the selectors, the dataset call, and enqueue_links are all untouched — Crawlee's queue and deduplication keep working, because only the object that fetches bytes was replaced. Keep the key in the environment as SCRAPELESS_API_KEY; the Universal Scraping API getting-started guide lists the other request parameters. If you need proxy routing for the default HTTP client instead, the Crawlee proxy guide covers that configuration.

Getting started takes a minute — create a free Scrapeless account and the free plan covers everything here.

Run It

export SCRAPELESS_API_KEY="your-api-key"
python3 crawlee_demo.py

The complete output from the verification run:

crawlee 1.9.0 | beautifulsoup4 4.15.0
purge_on_start default: True
--- static site, default HTTP client, isolated storage ---
  static
    requests finished : 3
    dataset items     : 30
    distinct pages    : 3
    first quote       : “The world as we have created it is a process of our think
    first author      : Albert Einstein
--- javascript site, default HTTP client, isolated storage ---
  javascript
    requests finished : 1
    dataset items     : 0
    distinct pages    : 0
--- javascript site, ScrapelessHttpClient, isolated storage ---
  javascript+api
    requests finished : 1
    dataset items     : 10
    distinct pages    : 1
    first quote       : “The world as we have created it is a process of our think
--- two crawlers, one process, default storage ---
  crawler A
    requests finished : 2
    dataset items     : 20
    distinct pages    : 2
  crawler B
    requests finished : 3
    dataset items     : 50
    distinct pages    : 5

Troubleshooting

The dataset has more items than this run produced. Storage is shared per process. Set Configuration(storage_dir=...) per crawler, or delete ./storage between runs, or run one crawler per process.

desired_concurrency cannot be greater than max_concurrency. ConcurrencySettings validates the pair on construction. Lowering max_concurrency alone raises; set desired_concurrency to match.

ModuleNotFoundError: No module named 'bs4'. The base package has no parser. Install crawlee[beautifulsoup] or crawlee[parsel].

ImportError on HttpHeaders. It is exported from the top-level crawlee package, not from a submodule.

Zero items and one finished request. The page is rendered client-side. Print await context.http_response.read() and search it for a value you can see on the page; if the value is absent, no selector will find it.

Conclusion

Crawlee's value is the machinery around your handler: a queue, deduplication, bounded concurrency, and a dataset. That machinery is also the thing to watch, because it holds state that survives the crawler object. The two measurements in this guide both come from that fact — a second crawler in one process reporting 50 items when it fetched far fewer, and a client-rendered page returning a clean zero.

Both are diagnosable in one line. Print requests_finished alongside the dataset count on every run; when they disagree with your configuration, look at the storage before the selectors. And when the count is zero because the markup arrived empty, the smallest fix is to change the transport and leave the handler alone.

Ready to try it? Start with the Scrapeless free plan and see the current pricing for higher volumes.

FAQ

Q: Which Crawlee crawler class should I start with?

Start with BeautifulSoupCrawler if the data is in the served HTML, because it costs one HTTP request per page and hands you a familiar parsed tree. Move to ParselCrawler if you prefer XPath, HttpCrawler if you want the raw bytes, and a Playwright crawler only when the page genuinely needs a browser.

Q: How is Crawlee different from writing the loop myself?

Crawlee supplies the request queue, URL deduplication, bounded concurrency, and dataset persistence. In the run above, enqueue_links(selector="li.next a") walked three pages without the handler constructing a single URL or tracking which pages it had seen.

Q: Why does my dataset contain results from an earlier run?

Because Crawlee's storage is shared per process and purge_on_start fires once when storage is first opened, not per crawler. Two crawlers in one script share a dataset and a request queue. Give each one a Configuration(storage_dir=...), or run one crawler per process.

Q: Do I have to switch to PlaywrightCrawler for JavaScript pages?

No. PlaywrightCrawler is one option, but it changes the crawler class and the context your handler receives. Implementing Crawlee's HttpClient interface changes only how bytes are fetched, which is why the handler in this guide went from 0 items to 10 without an edit.

Q: Where does Crawlee write its output?

Under ./storage by default, with datasets in storage/datasets/. crawler.get_data() reads the dataset back in the same process, and Configuration(storage_dir=...) moves the whole tree somewhere else.