Scrapy Web Scraping: Build a Spider That Handles JavaScript Pages

TL;DR:

• Scrapy is a crawling framework, not an HTTP client. It gives you a scheduler, a duplicate filter, an asynchronous downloader, and an item pipeline, so a spider stays about twenty lines even when it walks a hundred pages.
• A spider is one class with three required pieces: a name, a starting URL, and a parse method that yields dictionaries. Everything else is configuration.
• Scrapy never executes JavaScript. The same spider that returns 10 items from a server-rendered page returns 0 items from its client-rendered twin, because the HTML that arrives contains an empty container and a script tag.
• A downloader middleware fixes that without touching the spider. Rendering the page upstream and handing Scrapy back an ordinary HtmlResponse restored all 10 items while the parse method stayed byte-identical.
• Version pinning matters more than usual here. Scrapy's TLS layer sits on Twisted and pyOpenSSL, and two specific combinations fail every HTTPS download with errors that name certificates rather than dependencies.
• Start free. The Universal Scraping API used in the pivot has a free tier, so you can run the whole comparison in this post without a paid plan.

Scrapy separates the parts of a crawl you care about from the parts you do not. You write a selector. Scrapy handles the request queue, the concurrency, the deduplication, the encoding detection, and the serialization.

Then you point it at a page built by a front-end framework and get an empty file.

This guide builds a working spider, breaks it on purpose against a JavaScript-rendered page, and repairs it with a downloader middleware — the piece of Scrapy that lets you change how pages are fetched without changing how they are parsed.

What Scrapy Gives You That a Request Loop Does Not

Scrapy is a crawling engine with an opinion about structure. A hand-rolled loop over a list of URLs works until you need the things Scrapy already has:

• A scheduler with a duplicate filter. Requests are queued, deduplicated by fingerprint, and dispatched with bounded concurrency.
• Asynchronous downloads without async syntax. Scrapy runs on the Twisted reactor, so many requests are in flight while your parse method reads like ordinary synchronous code.
• Selectors built in. response.css() and response.xpath() come from Parsel, the same selector library covered in the CSS and XPath selector guide.
• Feed exports. -O results.json writes JSON, JSON Lines, CSV, or XML with no serialization code.
• Politeness settings. ROBOTSTXT_OBEY, DOWNLOAD_DELAY, and AUTOTHROTTLE_ENABLED are settings rather than something you implement. The first of those reads the file described in the Robots Exclusion Protocol standard.

The cost is that Scrapy has a shape you have to learn. The payoff arrives around the third page of a crawl.

Install Scrapy

Install into a fresh virtual environment and pin the TLS stack explicitly:

python3 -m venv .venv
source .venv/bin/activate
pip install "scrapy==2.17.0" "twisted==26.4.0" "pyopenssl==25.3.0"

Those three pins are deliberate. Scrapy's HTTPS support is layered on Twisted, which in turn calls pyOpenSSL, and the two failure modes documented at the end of this post both come from that stack rather than from Scrapy itself.

Confirm the versions before writing any spider code:

python3 -c "import importlib.metadata as m; print(m.version('scrapy'), m.version('twisted'), m.version('pyopenssl'))"

Write Your First Spider

A Scrapy spider is a class with a name, a list of starting URLs, and a parse method that receives a response and yields items. Save this as quotes_spider.py:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

response.css("div.quote") returns a selector list, so the loop iterates over elements rather than strings. ::text is Scrapy's pseudo-element for the text node, and .get() returns the first match while .getall() returns every match — which is why tags is a list and author is not. response.follow accepts the relative href directly, resolving it against the current URL, so there is no urljoin call.

The last block is pagination. Yielding a request from parse puts it back on the scheduler, and pointing its callback at parse walks the whole listing. The general shapes this pattern covers are laid out in the guide to pagination in web scraping.

Run It Without a Project

scrapy startproject generates a package with settings, pipelines, and a spiders directory. You do not need any of that yet. runspider executes a single file, and -s overrides any setting from the command line:

scrapy runspider quotes_spider.py -O quotes.json \
  -s LOG_LEVEL=ERROR \
  -s CLOSESPIDER_PAGECOUNT=3

-O truncates the output file, while -o appends to it — a distinction worth internalizing early. CLOSESPIDER_PAGECOUNT=3 bounds the crawl to three pages, which keeps a tutorial run polite and repeatable.

That command wrote 30 items: ten quotes per page across three pages, with the first record reading Albert Einstein and tags ['change', 'deep-thoughts', 'thinking', 'world'].

Twenty lines of spider, three pages crawled, pagination followed, JSON on disk. This is the part Scrapy is genuinely good at.

Where Scrapy Stops: JavaScript-Rendered Pages

Point the identical spider at the client-rendered twin of the same site by changing one line:

    start_urls = ["https://quotes.toscrape.com/js/"]

Then run it again:

scrapy runspider quotes_spider.py -O js.json -s LOG_LEVEL=ERROR -s CLOSESPIDER_PAGECOUNT=1

The result is an empty array. Zero items, no error, exit code 0.

Nothing is wrong with the selector. The page returned HTTP 200 and Scrapy parsed it correctly — the markup it received simply has no div.quote elements in it. The quotes are written into the DOM by a script after load, and script execution is a browser behavior defined by the HTML Standard's scripting model. Scrapy is an HTTP client with an HTML parser attached. It fetches bytes; it does not run a JavaScript engine, and Scrapy's own guidance on dynamically loaded content says so directly.

Watch for that silent zero. A crawl of a JavaScript page looks identical, in both output and exit code, to a crawl of a page with nothing on it.

The usual answers attach a browser: scrapy-playwright drives Chromium per request, and Splash runs a rendering service alongside the crawl. Both work, and both mean every request now carries a browser's memory and startup cost, plus a second runtime to deploy.

Moving the rendering off the machine entirely leaves Scrapy's request model untouched.

Render Upstream With a Downloader Middleware

A downloader middleware sits between Scrapy's engine and its downloader, and it has exactly the hook this problem needs. The behavior is specified: when process_request() returns a Request object, the downloader middleware reference states that "Scrapy will stop calling process_request() methods and reschedule the returned request." Its counterpart process_response() then returns a Response back up the chain.

So the middleware can swap each outgoing request for a POST to a rendering endpoint, then unwrap the reply into an ordinary HtmlResponse carrying the original URL. The spider never learns that anything happened.

Save this as scrapeless_middleware.py:

import json
import os

from scrapy.http import HtmlResponse

UNLOCKER = "https://api.scrapeless.com/api/v2/unlocker/request"


class ScrapelessMiddleware:
    """Render each request upstream, then hand Scrapy an ordinary HtmlResponse."""

    def __init__(self, token, country):
        self.token = token
        self.country = country

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            os.environ["SCRAPELESS_API_KEY"],
            crawler.settings.get("SCRAPELESS_PROXY_COUNTRY", "US"),
        )

    def process_request(self, request, spider):
        if request.meta.get("scrapeless"):
            return None
        payload = {
            "actor": "unlocker.webunlocker",
            "input": {
                "url": request.url,
                "proxy_country": self.country,
                "js_render": True,
            },
        }
        return request.replace(
            url=UNLOCKER,
            method="POST",
            body=json.dumps(payload),
            headers={"Content-Type": "application/json", "x-api-token": self.token},
            meta={**request.meta, "scrapeless": True, "origin_url": request.url},
            dont_filter=True,
        )

    def process_response(self, request, response, spider):
        if not request.meta.get("scrapeless"):
            return response
        rendered = json.loads(response.text)["data"]
        return HtmlResponse(
            url=request.meta["origin_url"],
            body=rendered,
            encoding="utf-8",
            request=request,
        )

The scrapeless flag in request.meta prevents infinite recursion. Without it, the rescheduled POST would re-enter process_request and be wrapped again. dont_filter=True is required because every rendered request now targets the same endpoint URL, and the duplicate filter would otherwise discard all but the first.

origin_url is what makes the swap invisible. The HtmlResponse is constructed with the page's real address rather than the API's, so response.url is correct and response.follow continues to resolve relative links against the right base. The request that reaches the wire is a POST, per the HTTP semantics specification, while the response the spider sees is an ordinary HTML document.

Finally, the API key is read from the SCRAPELESS_API_KEY environment variable inside from_crawler, so no credential is written into a settings file. Full parameter documentation lives in the Universal Scraping API reference, and the rendering behavior behind js_render is covered in the page rendering guide.

Wire It In and Run the Same Spider Again

Export the key, then enable the middleware with a setting. PYTHONPATH=. lets runspider import a module from the working directory:

export SCRAPELESS_API_KEY="your_api_key"

PYTHONPATH=. scrapy runspider quotes_spider.py -O js_unlocked.json \
  -s LOG_LEVEL=ERROR \
  -s CLOSESPIDER_PAGECOUNT=1 \
  -s 'DOWNLOADER_MIDDLEWARES={"scrapeless_middleware.ScrapelessMiddleware": 543}'

That run produced 10 items, with the first record again reading Albert Einstein and tags ['change', 'deep-thoughts', 'thinking', 'world'] — the same records the server-rendered page gives up for free.

quotes_spider.py changed by zero lines between that run and the failing one. The selectors, the pagination, the item shape, and the feed export all survived a complete change in how pages are fetched, which is the argument for putting rendering in a middleware instead of in the spider.

Getting started needs no card — the free plan covers a run this size.

Prove All Three Cases in One Script

Three separate commands are easy to run inconsistently. This script runs all three crawls in a single process and prints a comparison, so the claim above can be checked in one shot:

import json
import os

import scrapy
from scrapy import signals
from scrapy.crawler import CrawlerProcess
from scrapy.http import HtmlResponse

UNLOCKER = "https://api.scrapeless.com/api/v2/unlocker/request"


class ScrapelessMiddleware:
    """Render each request upstream, then hand Scrapy an ordinary HtmlResponse."""

    def __init__(self, token, country):
        self.token = token
        self.country = country

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            os.environ["SCRAPELESS_API_KEY"],
            crawler.settings.get("SCRAPELESS_PROXY_COUNTRY", "US"),
        )

    def process_request(self, request, spider):
        if request.meta.get("scrapeless"):
            return None
        payload = {
            "actor": "unlocker.webunlocker",
            "input": {"url": request.url, "proxy_country": self.country, "js_render": True},
        }
        return request.replace(
            url=UNLOCKER,
            method="POST",
            body=json.dumps(payload),
            headers={"Content-Type": "application/json", "x-api-token": self.token},
            meta={**request.meta, "scrapeless": True, "origin_url": request.url},
            dont_filter=True,
        )

    def process_response(self, request, response, spider):
        if not request.meta.get("scrapeless"):
            return response
        rendered = json.loads(response.text)["data"]
        return HtmlResponse(
            url=request.meta["origin_url"],
            body=rendered,
            encoding="utf-8",
            request=request,
        )


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def __init__(self, url, **kwargs):
        super().__init__(**kwargs)
        self.start_urls = [url]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }


class ScrapelessQuotesSpider(QuotesSpider):
    name = "quotes-scrapeless"
    custom_settings = {"DOWNLOADER_MIDDLEWARES": {ScrapelessMiddleware: 543}}


def main():
    import importlib.metadata as md

    print(
        "scrapy", md.version("scrapy"),
        "| twisted", md.version("twisted"),
        "| pyopenssl", md.version("pyopenssl"),
    )

    jobs = [
        ("static page, plain Scrapy", QuotesSpider, "https://quotes.toscrape.com/"),
        ("javascript page, plain Scrapy", QuotesSpider, "https://quotes.toscrape.com/js/"),
        ("javascript page, Scrapeless middleware", ScrapelessQuotesSpider, "https://quotes.toscrape.com/js/"),
    ]
    collected = {label: [] for label, _, _ in jobs}

    # Scrapy holds signal handlers weakly, so keep a strong reference to each one.
    handlers = []

    def collector(label):
        def on_item(item, response, spider):
            collected[label].append(item)
        handlers.append(on_item)
        return on_item

    process = CrawlerProcess({
        "LOG_LEVEL": "ERROR",
        "SCRAPELESS_PROXY_COUNTRY": "US",
    })
    for label, spider_cls, url in jobs:
        crawler = process.create_crawler(spider_cls)
        crawler.signals.connect(collector(label), signal=signals.item_scraped)
        process.crawl(crawler, url=url)
    process.start()

    for label, _, _ in jobs:
        items = collected[label]
        print(f"{label}: {len(items)} items")
        if items:
            print(f"  first author: {items[0]['author']}")
            print(f"  first tags: {items[0]['tags']}")

    print("parse method shared by both spiders:", ScrapelessQuotesSpider.parse is QuotesSpider.parse)


if __name__ == "__main__":
    main()

Running it prints:

scrapy 2.17.0 | twisted 26.4.0 | pyopenssl 25.3.0
static page, plain Scrapy: 10 items
  first author: Albert Einstein
  first tags: ['change', 'deep-thoughts', 'thinking', 'world']
javascript page, plain Scrapy: 0 items
javascript page, Scrapeless middleware: 10 items
  first author: Albert Einstein
  first tags: ['change', 'deep-thoughts', 'thinking', 'world']
parse method shared by both spiders: True

ScrapelessQuotesSpider subclasses QuotesSpider and adds nothing but custom_settings, which is why the final line is True: both crawls dispatched the same parse function object. The 0 in the middle is the JavaScript problem, and the 10 below it is the fix, measured against the identical parser.

custom_settings on a spider class scopes settings to that spider, which is what lets one process run crawls with and without the middleware. Scrapy also holds signal receivers by weak reference, so a collector closure created inline is garbage-collected before the crawl finishes and silently records nothing — the handlers list exists to hold them.

Troubleshooting

Every HTTPS request fails with 'X509' object has no attribute 'get_extension'. Twisted is too old for the installed pyOpenSSL. Twisted 24.3.0, which several Linux distributions still package, calls a method that current pyOpenSSL releases no longer provide. Installing twisted==26.4.0 clears it. This surfaces most often when Scrapy is installed into a system Python rather than a virtual environment.

Every HTTPS request fails with certificate verify failed. Scrapy 2.17.0 paired with Twisted 26.4.0 and pyOpenSSL 26.3.0 fails certificate verification on ordinary public sites. Pinning pyopenssl==25.3.0 resolves it, which is why the install step names all three versions.

The middleware never runs. runspider does not add the working directory to the import path, so the class named in DOWNLOADER_MIDDLEWARES cannot be imported. Prefix the command with PYTHONPATH=., or move the spider into a project created by scrapy startproject, where module resolution is handled for you.

Every request after the first is dropped. The duplicate filter is fingerprinting the rendering endpoint, which is identical for every page. dont_filter=True on the replacement request is what prevents this.

The spider yields items but response.follow builds wrong URLs. The HtmlResponse was constructed with the API endpoint as its URL instead of the original page address. Relative links resolve against response.url, so origin_url has to be carried through request.meta.

Conclusion

Scrapy's division of labor is what makes it worth the learning curve. The spider owns what data means; the downloader owns how bytes arrive. Keeping those separate is why a page that returned nothing could be made to return ten records without editing a single selector.

Build the spider first against whatever the server sends. When the selectors come back empty, check whether the markup ever contained them before reaching for a browser. If it did not, rendering upstream through a middleware keeps the crawl asynchronous, keeps deployment to one Python process, and leaves the parsing code you already tested exactly as it was.

Ready to run this against your own targets? Create a free Scrapeless account, export your key, and drop the middleware into an existing spider. See the Universal Scraping API product page for the rendering surface, and the pricing page for plan limits.

FAQ

Q: Can Scrapy scrape JavaScript-rendered websites on its own?

No. Scrapy fetches HTML over HTTP and parses it, with no JavaScript engine in the pipeline. A page that builds its content client-side arrives as an empty container plus a script tag, and selectors match nothing. Rendering has to happen somewhere else — in a browser attached to the crawl, or upstream in a rendering API whose output is fed back through a downloader middleware.

Q: What is the difference between a downloader middleware and a spider middleware?

A downloader middleware sits between the engine and the downloader, so it sees every request before it is sent and every response before it is parsed — the right place for proxies, headers, and rendering. A spider middleware sits between the engine and the spider, handling the items and requests your callbacks produce. Changing how a page is fetched belongs in a downloader middleware.

Q: Do I need scrapy startproject, or is a single file enough?

A single file run with scrapy runspider is enough for one spider, and every command in this guide uses it. Create a project once you need shared settings, item pipelines, several spiders, or deployment — the project layout gives you a settings module and import paths that resolve without PYTHONPATH.

Q: Why does my spider return zero items with no error message?

Because an empty selector match is not an error in Scrapy. The most common causes are content injected by JavaScript after load, a selector written against markup that a browser's inspector shows but the raw response does not contain, or a response that returned an interstitial page with HTTP 200. Print len(response.text) and search the body for a string you expect before assuming the selector is wrong.

Q: Does the middleware slow the crawl down?

Each request becomes a rendered fetch rather than a raw one, so per-request latency goes up. Scrapy's concurrency model is unchanged, though: requests still run through the same scheduler and the same CONCURRENT_REQUESTS limit, with no browser process per request. Enable the middleware only for the domains that need it and leave the rest on the plain downloader.

Q: How do I keep my API key out of the codebase?

Read it from the environment inside from_crawler, as the middleware here does with os.environ["SCRAPELESS_API_KEY"]. The key never appears in a settings file, so nothing sensitive is committed and the same code runs in development and production against different credentials.