Claude Web Scraping: Turn Nullable Stat Tables Into Clean JSON

TL;DR:

• Claude reads nullable data cleanly, and a live run proves it. Fed a rendered season-stats table, anthropic/claude-haiku-4.5 returned 25 team-season records in one call, correctly writing null for a stat that didn't exist yet in the 1990-91 NHL season instead of guessing a value.
• The API is text-in, text-out — it never touches the network. Rendering, sessions, and access challenges belong to a fetch layer; this guide's is one POST per page through the Scrapeless Universal Scraping API.
• JSON mode has a real quirk worth knowing before it costs you a parse error. Claude via OpenRouter sometimes wraps structured output in a ```json fence even with response_format set — the extraction script below strips it defensively.
• Current cheap-tier naming moves. The live OpenRouter model catalog puts claude-haiku-4.5 well above the older claude-3-haiku line in capability at a modest per-token premium; this guide checked the catalog directly rather than assuming a name from memory.
• No Anthropic key yet? The identical request runs through OpenRouter. This guide executed it live on anthropic/claude-haiku-4.5 and shows the captured output.
• Free to start on the fetch side. Create your Scrapeless API key at app.scrapeless.com.

Can Claude scrape websites?

Claude parses; it does not fetch. Hand it page text and a schema and it returns clean, typed records — including the harder case of knowing when a field is genuinely absent rather than inventing a plausible-looking zero. What it cannot do is retrieve a specific URL, execute the JavaScript that builds a page, hold a session, or clear an access challenge at the volume a scraping job needs. Every working "Claude web scraping" setup pairs the model with a fetch layer that does that separately.

An earlier guide on this site, Web Scraping with Claude AI, surveys ten different ways to combine Claude with a scraper. This guide is narrower and more concrete: one real target, one live-executed extraction, one schema, real captured output. If the broader question is which language model to use for parsing at all, the LLM scraper explainer covers the category.

Install

The Anthropic SDK covers the native path, openai covers the OpenRouter path, and requests covers the fetch layer:

pip install "anthropic==0.120.0" "openai==2.48.0" requests

Configure

export ANTHROPIC_API_KEY="sk-ant-your_key"
export SCRAPELESS_API_KEY="sk_your_scrapeless_key"

Fetch a page with a field that's sometimes missing

The demo target is a public sports-statistics sandbox: one season-by-season row per NHL team, including a stat — overtime losses — that the league did not track until partway through the 1990s. Rows from before that point leave the cell blank, which makes the page a genuine test of whether an extractor invents data or reports it missing. One POST to the Universal Scraping API returns the page with rendering and proxy routing handled server-side:

# fetch_teams.py — retrieve the team-stats page through Scrapeless
import os

import requests

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={
        "Content-Type": "application/json",
        "x-api-token": os.environ["SCRAPELESS_API_KEY"],
    },
    json={
        "actor": "unlocker.webunlocker",
        "input": {"url": "https://scrapethissite.com/pages/forms/", "method": "GET", "js_render": True},
    },
    timeout=120,
)
resp.raise_for_status()
html = resp.json().get("data", "")
print(f"fetched {len(html):,} characters")
print("team rows:", html.count('<tr class="team">'))

with open("page.html", "w", encoding="utf-8") as f:
    f.write(html)

The run prints 25 team rows for the sandbox's first page of results — a live target maintained by a public web-scraping practice site, distinct from the quotes and books sandboxes used in this site's other LLM-parsing guides.

Basic implementation: Claude as the extractor

The native Messages API takes a JSON Schema directly through output_config, which the current API reference documents as the replacement for the older prefill-based JSON tricks — see Anthropic's structured-outputs guide. The current cheapest non-legacy Claude tier is claude-haiku-4-5; note the native model ID uses hyphens, distinct from the claude-haiku-4.5 slug OpenRouter lists.

Note: This block needs an ANTHROPIC_API_KEY with credit — the one prerequisite this guide does not assume. The next section runs the identical extraction live through OpenRouter, with captured output.

# extract_claude.py — native Claude extraction (requires ANTHROPIC_API_KEY)
import json

from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment

page_html = open("page.html", encoding="utf-8").read()

response = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=6000,
    system="Extract every team-season row from this table. ot_losses is null when the cell is blank.",
    messages=[{"role": "user", "content": page_html}],
    output_config={
        "format": {
            "type": "json_schema",
            "schema": {
                "type": "object",
                "properties": {
                    "teams": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "name": {"type": "string"},
                                "year": {"type": "integer"},
                                "wins": {"type": "integer"},
                                "losses": {"type": "integer"},
                                "ot_losses": {"type": ["integer", "null"]},
                                "win_pct": {"type": "number"},
                                "goals_for": {"type": "integer"},
                                "goals_against": {"type": "integer"},
                            },
                            "required": ["name", "year", "wins", "losses", "ot_losses", "win_pct", "goals_for", "goals_against"],
                            "additionalProperties": False,
                        },
                    }
                },
                "required": ["teams"],
                "additionalProperties": False,
            },
        }
    },
)

text = next(b.text for b in response.content if b.type == "text")
data = json.loads(text)
print(f"extracted {len(data['teams'])} teams")

output_config.format guarantees the response parses as JSON against the schema — no fence-stripping, no prefill workaround.

No Anthropic key? Run it through OpenRouter

OpenRouter exposes Claude behind an OpenAI-compatible chat-completions surface. This is the version this guide executed for real, fetch and extraction in one self-contained script:

# extract_openrouter.py — the same extraction, executed via OpenRouter
import json
import os

import requests
from openai import OpenAI

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={
        "Content-Type": "application/json",
        "x-api-token": os.environ["SCRAPELESS_API_KEY"],
    },
    json={
        "actor": "unlocker.webunlocker",
        "input": {"url": "https://scrapethissite.com/pages/forms/", "method": "GET", "js_render": True},
    },
    timeout=120,
)
resp.raise_for_status()
page_html = resp.json().get("data", "")

client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"])

completion = client.chat.completions.create(
    model="anthropic/claude-haiku-4.5",
    temperature=0,
    max_tokens=6000,
    response_format={"type": "json_object"},
    messages=[
        {
            "role": "system",
            "content": 'Extract every team-season row from this table. Reply ONLY with JSON: '
            '{"teams":[{"name":str,"year":int,"wins":int,"losses":int,"ot_losses":int|null,'
            '"win_pct":number,"goals_for":int,"goals_against":int}]}. '
            "ot_losses is null when the cell is blank.",
        },
        {"role": "user", "content": page_html},
    ],
)

raw = completion.choices[0].message.content.strip()
if raw.startswith("```"):
    raw = raw.split("```")[1]
    if raw.startswith("json"):
        raw = raw[4:]
data = json.loads(raw)
print(f"extracted {len(data['teams'])} teams")
print(json.dumps(data["teams"][0], ensure_ascii=False))

The live run pulled all 25 team-seasons, first record:

extracted 25 teams
{"name": "Boston Bruins", "year": 1990, "wins": 44, "losses": 24, "ot_losses": null, "win_pct": 0.55, "goals_for": 299, "goals_against": 264}

ot_losses came back null for a 1990-91 season — the year predates the NHL tracking that statistic — and every one of the 25 rows parsed to the declared types with no post-hoc casting. The fence-stripping matters here: without it, json.loads fails outright on this model's OpenRouter output, even with response_format set. The whole call: 13,365 tokens.

Get your API key on the free plan: app.scrapeless.com

Advanced patterns

• Defend against fenced output on the aggregator path. The native output_config.format path enforces schema-true JSON with no fences; the OpenRouter chat-completions path does not make that same guarantee model-to-model, so strip a leading ``` block before parsing regardless of which model you route to.
• Name nullability explicitly. The schema alone ("ot_losses": {"type": ["integer", "null"]}) plus one sentence in the system prompt is what produced correct nulls instead of a model inventing 0. Leave either out and re-test before trusting the field.
• Keep one page per call. Page boundaries are natural record boundaries; batching several pages into one prompt blurs where one team's stats end and the next begins.
• Measure tokens before scaling up. The run above cost 13,365 tokens for a 25-row page — multiply by real page and field counts before committing to a page volume, not after the bill arrives.

Troubleshooting

• json.loads fails on an OpenRouter response even with JSON mode set. Check for a leading ```json fence and strip it before parsing — this guide's live run needed exactly that fix.
• A field that should sometimes be missing always comes back as 0 or a placeholder. State explicitly in the schema and the prompt when a field is nullable and what condition makes it null; models default to filling gaps unless told not to.
• Zero or a handful of rows from a page that has dozens. Check the fetched HTML count first, the way the fetch script above does — a thin fetch is a rendering or access problem, not something the extraction prompt can fix.
• Output drifts between identical runs. Set temperature=0 on the OpenRouter path; extraction is a transcription task, and any temperature above zero invites paraphrasing.

Conclusion

Claude's structured-output path removes the guesswork from getting typed JSON back — output_config.format on the native API skips prefill tricks entirely, and even the more permissive OpenRouter chat-completions surface got all 25 team-season rows out of a real page with correctly nulled fields, once the response was defended against markdown fences. What Claude does not supply is the page itself: one server-side POST per page, through a fetch layer built for rendering and access, is what makes the records exist to extract from in the first place.

Ready to Feed Claude Real Pages?

The fetch layer here is the Universal Scraping API — plans and request volumes are on the pricing page, with every unlocker.webunlocker parameter in the developer docs. Create a key on the free plan at app.scrapeless.com and both scripts run as written.

FAQ

Q: Can Claude scrape websites by itself?

No. The Claude API is a language-model endpoint: it extracts from text you provide and cannot issue HTTP requests, render JavaScript, or hold a session. Every working setup pairs it with a fetch layer that returns faithful page content.

Q: Which Claude model should I use for web-scraping extraction?

claude-haiku-4-5 on the native API (claude-haiku-4.5 on OpenRouter) — the live run in this guide used it and returned all 25 records with correctly nulled fields. It is the current cheapest non-legacy Claude tier; the older claude-3-haiku name is a prior generation still listed but not the current recommendation.

Q: Why does my Claude JSON output fail to parse even with JSON mode on?

On the OpenRouter aggregator path, Claude can wrap its output in a ```json markdown fence even when response_format is set to json_object. Strip a leading fence before calling json.loads, or use the native API's output_config.format, which enforces schema-true JSON without that failure mode.

Q: How does Claude handle a field that's sometimes missing from the source page?

Correctly, when the schema and prompt say so. This guide's schema marked ot_losses as ["integer", "null"] and the system prompt stated the null condition; the live run returned null for a season that predates the statistic rather than fabricating a value.

Q: Is scraping with Claude legal?

The extraction layer does not change collection rules. Fetch public pages only, respect site terms and the robots directives standardized by the Robots Exclusion Protocol, keep volumes bounded, and handle any personal data under applicable law — plus Anthropic's usage policies on the model side.