How to Build a Self-Healing Web Scraper

TL;DR:

• A CSS selector that works today breaks the morning a site renames a class, and a normal scraper responds by silently returning nothing.
• A self-healing scraper repairs the selector at runtime: it fetches the page, tries the saved selector, and on a miss asks a language model for a new one read from the live DOM.
• This pipeline fetches the rendered page through the Scrapeless Universal Scraping API, so the model always heals against the real, script-executed markup rather than an empty shell.
• A live run shows the saved selector .author-name matching zero elements, then deepseek-v4-flash proposing small.author, which extracts all ten authors.
• The healed selector is validated against the DOM before it is trusted — a wrong guess matches nothing and is discarded, so a bad suggestion never silently corrupts your data.
• Get a Scrapeless API key on the free plan and run the whole loop end to end.

Pipeline at a Glance

A scraper breaks not because the code is wrong but because the page moved. The fix is to treat the selector as data that can be regenerated, not a constant baked into the source. This pipeline does that in five stages:

fetch the rendered page (Scrapeless) → try the saved selector → detect the miss → heal with a language model → validate and extract → persist the working selector

Each stage below runs against the live page. The fetch and the heal are the two calls that leave your machine; everything else is local parsing you can inspect.

Stage 1: Fetch the rendered page with Scrapeless

The heal is only as good as the HTML you show the model, so the fetch has to return the page a browser would build, not the empty container a client-rendered site sends first. That rendered fetch is the job of the Scrapeless Universal Scraping API. The Universal Scraping API renders the page server-side with js_render and returns the finished HTML in its data field. Install the two libraries the pipeline uses:

pip install requests beautifulsoup4

Put both keys in the environment — the Scrapeless key for the fetch, the model key for the heal:

export SCRAPELESS_API_KEY="your_scrapeless_api_key"
export OPENROUTER_API_KEY="your_openrouter_api_key"

Stage 2: Try the saved selector and detect the break

Run last month's scraper against today's page and the failure is quiet: the selector matches nothing and the extract returns an empty list, so a naive job writes zero rows and reports success. The demo page below builds its quotes with JavaScript; the saved selector .author-name is a class the markup no longer uses, standing in for any selector a redesign has renamed:

import os, requests
from bs4 import BeautifulSoup

def fetch(url):
    r = requests.post(
        "https://api.scrapeless.com/api/v2/unlocker/request",
        headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"], "Content-Type": "application/json"},
        json={"actor": "unlocker.webunlocker", "input": {"url": url, "js_render": True}},
        timeout=120,
    )
    r.raise_for_status()
    return r.json()["data"]

soup = BeautifulSoup(fetch("https://quotes.toscrape.com/js/"), "html.parser")

SAVED_SELECTOR = ".author-name"          # worked last month; the site renamed the class
authors = [e.get_text(strip=True) for e in soup.select(SAVED_SELECTOR)]
print(f"saved selector {SAVED_SELECTOR!r} matched: {len(authors)}",
      "-> BROKEN, extracted nothing" if not authors else "-> ok")

The run makes the silent failure loud:

saved selector '.author-name' matched: 0 -> BROKEN, extracted nothing

Stage 3: Heal with a language model

When the saved selector matches nothing, hand the model the live DOM and ask for a replacement. The model reads the actual structure — the tags, classes, and nesting defined by the CSS Selectors specification — and returns a selector shaped to the page in front of it, not to a page it saw in training. Constraining the reply to a JSON object keeps the answer a single selector string rather than an explanation:

import os, json, requests

def propose_selector(html_fragment, target):
    prompt = (f'A web-scraping CSS selector broke. From this HTML fragment, return JSON '
              f'{{"selector": "<css>"}} for the element holding the {target}. Return ONLY JSON.\n\n'
              f"HTML:\n{html_fragment}")
    r = requests.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}", "Content-Type": "application/json"},
        json={"model": "deepseek/deepseek-v4-flash",
              "messages": [{"role": "user", "content": prompt}],
              "response_format": {"type": "json_object"}, "temperature": 0},
        timeout=120,
    )
    r.raise_for_status()
    return json.loads(r.json()["choices"][0]["message"]["content"])["selector"]

Sending one rendered .quote block is enough context and keeps the token count — and the cost — small.

Get your API key on the free plan: app.scrapeless.com

Stage 4: Validate and extract

A proposed selector is a suggestion until the DOM confirms it. Run the healed selector against the same soup and count the matches: a real fix returns rows, and a bad guess returns zero and is discarded before it can write empty data. The full loop ties the three calls together:

import os, json, requests
from bs4 import BeautifulSoup

def fetch(url):
    r = requests.post(
        "https://api.scrapeless.com/api/v2/unlocker/request",
        headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"], "Content-Type": "application/json"},
        json={"actor": "unlocker.webunlocker", "input": {"url": url, "js_render": True}},
        timeout=120,
    )
    r.raise_for_status()
    return r.json()["data"]

def propose_selector(html_fragment, target):
    prompt = (f'A web-scraping CSS selector broke. From this HTML fragment, return JSON '
              f'{{"selector": "<css>"}} for the element holding the {target}. Return ONLY JSON.\n\n'
              f"HTML:\n{html_fragment}")
    r = requests.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}", "Content-Type": "application/json"},
        json={"model": "deepseek/deepseek-v4-flash",
              "messages": [{"role": "user", "content": prompt}],
              "response_format": {"type": "json_object"}, "temperature": 0},
        timeout=120,
    )
    r.raise_for_status()
    return json.loads(r.json()["choices"][0]["message"]["content"])["selector"]

def extract(soup, selector):
    return [e.get_text(strip=True) for e in soup.select(selector)]

soup = BeautifulSoup(fetch("https://quotes.toscrape.com/js/"), "html.parser")
authors = extract(soup, ".author-name")

if not authors:
    print("saved selector '.author-name' matched: 0 -> healing")
    sample = str(soup.select_one(".quote") or soup.body)[:1500]
    healed = propose_selector(sample, "quote author name")
    print("model proposed selector:", repr(healed))
    authors = extract(soup, healed)
    print("healed selector matched:", len(authors))

print("authors:", ", ".join(authors[:3]), "...")

The broken selector heals and the extract fills again:

saved selector '.author-name' matched: 0 -> healing
model proposed selector: 'small.author'
healed selector matched: 10
authors: Albert Einstein, J.K. Rowling, Jane Austen ...

Because the healed selector is checked against the CSS selector engine that your parser already uses, the validation is exact: either it matches elements or it does not, and only a match is accepted.

Stage 5: Persist the working selector

Healing once is repair; healing every run is waste. After a selector validates, write it back to a small cache keyed by field, so the next run reads the working selector straight from disk and only calls the model when that one breaks too:

import json
from pathlib import Path

CACHE = Path("selectors.json")

def remember(field, selector):
    cache = json.loads(CACHE.read_text()) if CACHE.exists() else {}
    cache[field] = selector
    CACHE.write_text(json.dumps(cache, indent=2))

remember("quote_author", "small.author")   # next run starts from the healed selector

The scraper now degrades gracefully: it leans on cached selectors while they work and repairs them the moment a page shifts, instead of failing until someone notices and edits the code.

Conclusion

A self-healing scraper turns a broken selector from an outage into a runtime event. The Scrapeless Universal Scraping API supplies the rendered HTML the model needs to reason about, the model reads that live DOM and proposes a selector, and a match count against the parser decides whether the fix is real before any data is written. The run above — .author-name matching zero, small.author matching ten — is the loop in miniature: fetch, detect, heal, validate, persist. Size the request volume you expect against the pricing page before you scale, and read the primer on how a browser parses HTML into a DOM if you want to understand why the rendered fetch matters. For a wider view of language models in extraction, the explainer on what an LLM scraper is sets the context.

Join our community to claim a free plan and compare notes with other developers building resilient scrapers: Discord · Telegram.

FAQ

Q: What actually breaks when a scraper "breaks"?

Usually the selector, not the logic. A site ships a redesign that renames a class or restructures its markup, the saved CSS selector stops matching, and the extract returns an empty list. The request still succeeds, which is why the failure is easy to miss.

Q: Why fetch through the Scrapeless Universal Scraping API instead of a plain request?

Because the model can only heal against the HTML it is shown. A client-rendered page returns an empty container to a bare HTTP client, so the model would reason about markup that has no data in it. The Universal Scraping API renders the page server-side and returns the finished DOM, giving the model the real structure to work from.

Q: How do you stop the model from inventing a selector that matches nothing?

You validate before you trust. The healed selector is run against the parsed DOM and the matches are counted; a selector that returns zero elements is discarded rather than used. The model proposes, but the match count decides.

Q: Does this call the model on every run?

No. Once a selector validates, it is cached to disk keyed by field, and later runs read the working selector directly. The model is called only when a cached selector stops matching, which keeps both latency and cost tied to actual breakage rather than to every scrape.

Q: Is it legal to scrape this way?

The healing loop does not change what you are allowed to collect. Scrape public pages, honor the site's terms and its robots directives, avoid personal data you have no basis to process, and keep request rates modest.