Validating Scraped Data With Pydantic: Catch the Bugs Your Selectors Hide
Senior Web Scraping Engineer
TL;DR:
- A scraper that returns data is not a scraper that returns correct data. Twenty records collected live passed a permissive Pydantic model with every stock count silently set to
0. - The dangerous line is the fallback, not the parse.
return int(match.group(1)) if match else 0looks careful and is how wrong numbers reach a database. A validator that raises instead turned those 20 silent passes into 20 explicit rejections. - The rejections named the real bug. The listing page carries the string
In stockwith no number at all — the count only exists on each product's own page, which the extractor was never reading. - Validation converts types as well as checking them.
'£51.77'becomes aDecimal,'Three'becomes3, and'In stock (22 available)'becomes22, so downstream code never re-parses a string. - Field constraints catch what validators miss.
min_length,gt, andlerejected an empty title, a non-numeric price, and an out-of-range rating without a line of custom code. - Start free. The fetch stage runs on the Universal Scraping API's free tier.
A scraper that crashes tells you it broke. A scraper that fills a table with confidently wrong data tells you nothing, because every field had the right type and nobody checked whether the values meant anything.
This pipeline collects a live catalogue page, models the records with Pydantic, and then tightens the model until it finds a defect that was already there.
Pipeline at a Glance
fetch the page → extract raw strings → validate and coerce → tighten until it fails → fix the source
The interesting stage is the fourth one. The first three are ordinary, and they are what produced the bug.
Stage 1: Fetch the Page
Pydantic has no HTTP client and no HTML parser — it validates Python objects, so something has to hand it some. This pipeline fetches through the Universal Scraping API, which returns the rendered document as a string:
python
import json
import os
import urllib.request
UNLOCKER = "https://api.scrapeless.com/api/v2/unlocker/request"
def fetch(url):
payload = json.dumps({
"actor": "unlocker.webunlocker",
"input": {"url": url, "proxy_country": "US", "js_render": False},
}).encode()
request = urllib.request.Request(
UNLOCKER, data=payload,
headers={"Content-Type": "application/json",
"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
)
with urllib.request.urlopen(request, timeout=120) as response:
return json.loads(response.read().decode())["data"]The key stays in the SCRAPELESS_API_KEY environment variable, never in the file.
Stage 2: Extract Raw Strings
Everything the page gives you is text. That is the whole problem:
python
from bs4 import BeautifulSoup
def listing_records(html):
soup = BeautifulSoup(html, "html.parser")
return [{
"title": pod.h3.a["title"],
"price_gbp": pod.select_one("p.price_color").get_text(strip=True),
"in_stock": pod.select_one("p.instock.availability").get_text(strip=True),
"rating": pod.select_one("p.star-rating")["class"][1],
} for pod in soup.select("article.product_pod")]One record from a live run looks like this:
text
{"title": "A Light in the Attic", "price_gbp": "£51.77", "in_stock": "In stock", "rating": "Three"}A price with a currency symbol, an availability sentence, and a rating spelled as an English word taken from a CSS class. Nothing here can be summed, sorted, or compared until it is converted.
Stage 3: Validate and Coerce
A Pydantic model declares the types you want and the conversions that get you there. Type annotations are the declaration mechanism, following the PEP 484 type-hint specification, and a field_validator running in before mode transforms the raw value on the way in — the hook documented in the Pydantic validators reference:
python
import re
from decimal import Decimal
from pydantic import BaseModel, field_validator
RATING_WORDS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
class LenientBook(BaseModel):
title: str
price_gbp: Decimal
in_stock: int
rating: int
@field_validator("price_gbp", mode="before")
@classmethod
def parse_price(cls, value):
match = re.search(r"\d+\.\d{2}", value) if isinstance(value, str) else None
return Decimal(match.group()) if match else value
@field_validator("in_stock", mode="before")
@classmethod
def parse_availability(cls, value):
if not isinstance(value, str):
return value
match = re.search(r"\((\d+) available\)", value)
return int(match.group(1)) if match else 0 # <-- the silent fallback
@field_validator("rating", mode="before")
@classmethod
def parse_rating(cls, value):
return RATING_WORDS.get(value, value) if isinstance(value, str) else valueprice_gbp is a Decimal rather than a float deliberately. Binary floating point cannot represent most decimal fractions exactly, which is why the Python decimal module reference exists; money that will be summed or compared belongs in Decimal.
Run that model over the 20 live records and it reports complete success: 20 valid, 0 rejected.
It is wrong.
Stage 4: Tighten Until It Fails
Look again at the marked line. When the availability string does not match, the validator returns 0. That satisfies in_stock: int perfectly — 0 is an integer, the model is happy, and the record is written.
Replace each fallback with an exception, and add the constraints that describe what a valid record actually is:
python
from pydantic import BaseModel, Field, field_validator
class StrictBook(BaseModel):
title: str = Field(min_length=1)
price_gbp: Decimal = Field(gt=0)
in_stock: int = Field(ge=0)
rating: int = Field(ge=1, le=5)
@field_validator("price_gbp", mode="before")
@classmethod
def parse_price(cls, value):
if not isinstance(value, str):
return value
match = re.search(r"\d+\.\d{2}", value)
if not match:
raise ValueError(f"no price in {value!r}")
return Decimal(match.group())
@field_validator("in_stock", mode="before")
@classmethod
def parse_availability(cls, value):
if not isinstance(value, str):
return value
match = re.search(r"\((\d+) available\)", value)
if not match:
raise ValueError(f"no availability count in {value!r}")
return int(match.group(1))
@field_validator("rating", mode="before")
@classmethod
def parse_rating(cls, value):
if not isinstance(value, str):
return value
if value not in RATING_WORDS:
raise ValueError(f"unknown rating {value!r}")
return RATING_WORDS[value]The constraints come from the Pydantic field customization reference and cost nothing to add: a title must have characters, a price must be positive, a rating must sit between 1 and 5.
The same 20 records now produce 0 valid and 20 rejected, every one with the same message:
text
field=in_stock msg=Value error, no availability count in 'In stock'That is not the model being fussy. The listing page genuinely does not publish a stock count — it says In stock and nothing more. The permissive model had been recording a real number, 0, for a quantity the page never stated.
Stage 5: Fix the Source
The count exists, just not where the extractor was looking. Each product's own page carries it:
python
def detail_record(html):
soup = BeautifulSoup(html, "html.parser")
return {
"title": soup.h1.get_text(strip=True),
"price_gbp": soup.select_one("p.price_color").get_text(strip=True),
"in_stock": soup.select_one("p.instock.availability").get_text(strip=True),
"rating": soup.select_one("p.star-rating")["class"][1],
}On the detail page the same selector returns 'In stock (22 available)', and StrictBook accepts it — in_stock=22, price_gbp=51.77 as a Decimal, rating=3 as an int.
The fix was a change to the pipeline, not to the model. That is the point: the model's job was to refuse to guess, and refusing is what surfaced the missing page.
Getting started needs no card — the free plan covers a run this size.
The Whole Pipeline
python
import json
import os
import re
import urllib.request
from decimal import Decimal
from bs4 import BeautifulSoup
from pydantic import BaseModel, Field, ValidationError, field_validator
UNLOCKER = "https://api.scrapeless.com/api/v2/unlocker/request"
LISTING = "https://books.toscrape.com/"
DETAIL = "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
RATING_WORDS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
def fetch(url):
payload = json.dumps({
"actor": "unlocker.webunlocker",
"input": {"url": url, "proxy_country": "US", "js_render": False},
}).encode()
request = urllib.request.Request(
UNLOCKER, data=payload,
headers={"Content-Type": "application/json",
"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
)
with urllib.request.urlopen(request, timeout=120) as response:
return json.loads(response.read().decode())["data"]
class LenientBook(BaseModel):
title: str
price_gbp: Decimal
in_stock: int
rating: int
@field_validator("price_gbp", mode="before")
@classmethod
def parse_price(cls, value):
match = re.search(r"\d+\.\d{2}", value) if isinstance(value, str) else None
return Decimal(match.group()) if match else value
@field_validator("in_stock", mode="before")
@classmethod
def parse_availability(cls, value):
if not isinstance(value, str):
return value
match = re.search(r"\((\d+) available\)", value)
return int(match.group(1)) if match else 0 # <-- the silent fallback
@field_validator("rating", mode="before")
@classmethod
def parse_rating(cls, value):
return RATING_WORDS.get(value, value) if isinstance(value, str) else value
class StrictBook(BaseModel):
title: str = Field(min_length=1)
price_gbp: Decimal = Field(gt=0)
in_stock: int = Field(ge=0)
rating: int = Field(ge=1, le=5)
@field_validator("price_gbp", mode="before")
@classmethod
def parse_price(cls, value):
if not isinstance(value, str):
return value
match = re.search(r"\d+\.\d{2}", value)
if not match:
raise ValueError(f"no price in {value!r}")
return Decimal(match.group())
@field_validator("in_stock", mode="before")
@classmethod
def parse_availability(cls, value):
if not isinstance(value, str):
return value
match = re.search(r"\((\d+) available\)", value)
if not match:
raise ValueError(f"no availability count in {value!r}")
return int(match.group(1))
@field_validator("rating", mode="before")
@classmethod
def parse_rating(cls, value):
if not isinstance(value, str):
return value
if value not in RATING_WORDS:
raise ValueError(f"unknown rating {value!r}")
return RATING_WORDS[value]
def listing_records(html):
soup = BeautifulSoup(html, "html.parser")
return [{
"title": pod.h3.a["title"],
"price_gbp": pod.select_one("p.price_color").get_text(strip=True),
"in_stock": pod.select_one("p.instock.availability").get_text(strip=True),
"rating": pod.select_one("p.star-rating")["class"][1],
} for pod in soup.select("article.product_pod")]
def detail_record(html):
soup = BeautifulSoup(html, "html.parser")
return {
"title": soup.h1.get_text(strip=True),
"price_gbp": soup.select_one("p.price_color").get_text(strip=True),
"in_stock": soup.select_one("p.instock.availability").get_text(strip=True),
"rating": soup.select_one("p.star-rating")["class"][1],
}
def run(model, records):
valid, errors = [], []
for record in records:
try:
valid.append(model(**record))
except ValidationError as exc:
errors.append(exc)
return valid, errors
def main():
records = listing_records(fetch(LISTING))
print(f"listing records extracted: {len(records)}")
print(f"raw sample: {json.dumps(records[0], ensure_ascii=False)}")
lenient, lenient_errors = run(LenientBook, records)
zeroed = sum(1 for book in lenient if book.in_stock == 0)
print(f"lenient model: {len(lenient)} valid, {len(lenient_errors)} rejected")
print(f" silently zeroed in_stock: {zeroed} of {len(lenient)}")
strict, strict_errors = run(StrictBook, records)
print(f"strict model: {len(strict)} valid, {len(strict_errors)} rejected")
first = strict_errors[0].errors()[0]
print(f" first error: field={first['loc'][0]} msg={first['msg']}")
detail = detail_record(fetch(DETAIL))
print(f"detail raw availability: {detail['in_stock']!r}")
book = StrictBook(**detail)
print(f"detail validates: title={book.title!r} price_gbp={book.price_gbp} "
f"in_stock={book.in_stock} rating={book.rating}")
print(f" types: price={type(book.price_gbp).__name__} "
f"in_stock={type(book.in_stock).__name__} rating={type(book.rating).__name__}")
malformed = [
{**detail, "price_gbp": "Price on request"},
{**detail, "rating": "Eleven"},
{**detail, "title": ""},
]
print("malformed inputs:")
for record in malformed:
try:
StrictBook(**record)
print(" UNEXPECTEDLY VALID")
except ValidationError as exc:
err = exc.errors()[0]
print(f" rejected field={err['loc'][0]} type={err['type']}")
if __name__ == "__main__":
main()Its output:
text
listing records extracted: 20
raw sample: {"title": "A Light in the Attic", "price_gbp": "£51.77", "in_stock": "In stock", "rating": "Three"}
lenient model: 20 valid, 0 rejected
silently zeroed in_stock: 20 of 20
strict model: 0 valid, 20 rejected
first error: field=in_stock msg=Value error, no availability count in 'In stock'
detail raw availability: 'In stock (22 available)'
detail validates: title='A Light in the Attic' price_gbp=51.77 in_stock=22 rating=3
types: price=Decimal in_stock=int rating=int
malformed inputs:
rejected field=price_gbp type=value_error
rejected field=rating type=value_error
rejected field=title type=string_too_shortThe last three lines show the constraints working on inputs the validators alone would not catch. A non-numeric price and an unknown rating word are stopped by the validators that raise; the empty title is stopped by min_length=1, with the machine-readable code string_too_short and no custom code at all.
What Validation Can and Cannot Tell You
A model proves that a value has the right type and satisfies the rules you wrote. It cannot prove the value is true.
in_stock=0 was a perfectly valid integer for all 20 records. What made it wrong was that the page never said zero, and no type system can see that. The rule that catches it is narrower: this field must be derived from a pattern that was actually present. Writing validators that raise rather than substitute is how you encode it.
That distinction also decides where validation belongs. Run it at the boundary, on each record as it is produced, so a rejection points at one page. Aggregate checks over a finished dataset — the sort a dataframe is good at, as in the pandas pipeline guide — catch different problems and catch them later, once the offending page is hard to identify.
If your scraper runs inside a framework, mount the model where records leave the parser. In Scrapy that is an item pipeline, which sees every item from every spider before anything is written.
Troubleshooting
Every record is rejected after a site redesign. The selector still matches something, just not what it used to. Print the raw dict before validation — the error message quotes the offending value, which usually identifies the wrong element immediately.
A validator never runs. mode="before" validators run on raw input; without it a validator runs after Pydantic has already tried to coerce, so a string like '£51.77' fails on the type conversion before your code sees it.
Decimal comparisons behave oddly against floats. Do not mix them. Keep money in Decimal end to end and convert only at the display boundary.
A field is absent by design on some pages. Model it as int | None = None rather than defaulting to 0. None records the absence; 0 invents a measurement.
Rejected records vanish. Collect the ValidationError alongside the raw dict and write both somewhere. Rejections are the highest-value output of the run — they are a list of pages whose structure changed.
Conclusion
Validation earns its place when it fails. A model that passes everything you feed it is decoration; this one rejected 20 of 20 live records and, in doing so, reported that the pipeline had been reading a stock count off a page that does not publish one.
The habit worth taking is small: when a parse does not match, raise instead of substituting. A default value converts an obvious failure into a silent one, and silent failures are the ones that reach production.
Ready to put this on your own pipeline? Create a free Scrapeless account, export your key, and model one record type end to end before scaling the crawl. Plan limits are on the pricing page.
FAQ
Q: Why use Pydantic instead of writing my own checks?
Because the checks come with the type declaration rather than being scattered through the scraper. One model states what a valid record is, converts raw strings into usable types, and produces machine-readable errors with a field name and an error code. Hand-rolled checks drift out of sync with the code that writes the data; a model cannot, because nothing gets constructed without passing it.
Q: When should I use a field_validator and when a Field constraint?
Use a constraint for anything expressible as a rule about the finished value — min_length, gt, le, a pattern. Use a validator when the raw input has to be transformed before it can be checked, which for scraped data is most string fields. The two compose: the validator converts '£51.77' to a Decimal, then gt=0 checks the result.
Q: What should I do with records that fail validation?
Write them to a rejects file with the raw dict and the error, and keep the run going. A rejection is evidence about a page, not a reason to stop. Alerting on the rejection rate is more useful than alerting on individual failures, because a jump from a handful to everything means the site changed.
Q: Does this replace checking that my selectors work?
No, it catches a different failure. A selector test tells you whether an element was found; validation tells you whether what it contained makes sense. The bug in this post passed a selector test — p.instock.availability matched on every record — and was only caught by asking what the matched text actually said.
Q: Is Pydantic fast enough for large crawls?
Its validation core is compiled rather than pure Python, and for scraping the cost is dominated by network time anyway — one page fetch is orders of magnitude slower than validating the records it yields. If a profile ever shows otherwise, validate in batches rather than dropping the check.
Q: Can I generate the model from the data instead of writing it?
You can, and for exploration it is useful, but a model inferred from one sample encodes whatever that sample happened to contain. The value here came from writing down what a record should be and letting reality disagree — an inferred model would have learned that in_stock is always 0 and never objected.
At Scrapeless, we only access publicly available data while strictly complying with applicable laws, regulations, and website privacy policies. The content in this blog is for demonstration purposes only and does not involve any illegal or infringing activities. We make no guarantees and disclaim all liability for the use of information from this blog or third-party links. Before engaging in any scraping activities, consult your legal advisor and review the target website's terms of service or obtain the necessary permissions.
