How to Deduplicate Scraped Data With Entity Resolution

TL;DR:

• Four listing pages returned 71 records that resolved to 66 entities, with 5 genuine duplicate groups — the same books appearing in both a category listing and the paginated catalogue.
• Normalising before comparing is what makes an exact key work: case folding, Unicode normalisation and punctuation stripping turn three renderings of one title into one key.
• Blocking is measurable, not vague. 66 entities are 2,145 all-pairs comparisons; a four-character block key cuts that to 154 across 48 blocks, a 92.8% reduction.
• Pick the scorer before the threshold. The same pair of titles scored 87.8 on ratio and 100.0 on token_set_ratio.
• The threshold band is measurable on your own data: unrelated live titles peaked at 63.4 while three renderings of one book scored 93.5 and above.
• On a single clean catalogue, exact keys caught every duplicate and fuzzy matching found nothing above 90 — fuzzy earns its place across sources, not within one.
• Collect the multi-source records this reconciles with the Scrapeless free plan.

Scrape one site and duplicates are rare. Scrape the same catalogue through a category listing and a paginated index, or the same products across two retailers, and the duplicates arrive by construction — the crawl visited the same item by two routes and had no way to know.

Entity resolution is the step that turns those records back into things. It runs after extraction and before storage, and it is mostly a sequence of cheap decisions: what counts as the same string, what counts as the same record, and which version survives.

Every number below comes from one collection of 71 records taken from four live listing pages.

Pipeline at a Glance

The flow is normalise → exact key → block → fuzzy compare → merge. Each stage is cheaper than the one after it, so each one exists to reduce the work the next has to do.

Stage 1: Normalise Before Comparing

Two records describing the same product rarely carry byte-identical strings. Case, accents, and punctuation all move.

import re
import unicodedata

def norm(text):
    text = unicodedata.normalize("NFKD", text or "").casefold()
    text = re.sub(r"[^a-z0-9 ]+", " ", text)
    return re.sub(r"\s+", " ", text).strip()

The NFKD pass matters more than it looks. The Unicode normalisation annex defines several forms, and a compatibility decomposition is what makes a pre-composed é and a bare e plus combining accent compare equal — the two spellings are visually identical and byte-different, which is exactly the case that produces a duplicate nobody can see in the output.

Case folding rather than lowercasing is the matching counterpart, and the W3C character-model note on normalisation is the reference for why the two differ for non-ASCII text.

On the collected records:

[1] collected 71 records from 4 listing pages
    raw distinct titles        66
    normalised distinct titles 66

Identical here, because this catalogue is clean. That is worth knowing rather than assuming — running the comparison tells you whether normalisation is doing any work on your data before you build anything on top of it.

Stage 2: Group on an Exact Key

With a normalised key, the first pass is a grouping, not a comparison. It is O(n) and it catches every duplicate that agrees exactly.

from collections import defaultdict

by_key = defaultdict(list)
for record in records:
    by_key[norm(record["title"])].append(record)

dupe_groups = {k: v for k, v in by_key.items() if len(v) > 1}

[2] exact-key duplicates: 5 group(s), 10 records collapse to 5
    Sharp Objects                        x2  ['mystery', 'catalogue1']
    In a Dark, Dark Wood                 x2  ['mystery', 'catalogue2']
    In Her Wake                          x2  ['catalogue2', 'thriller']
    The Elephant Tree                    x2  ['catalogue2', 'thriller']
    Behind Closed Doors                  x2  ['catalogue2', 'thriller']

Note where the duplicates come from: every group spans two different listing pages. No single page contained a duplicate. That is the general shape — duplicates are a property of the crawl, not of the page, so a scraper that only ever reads one listing will not see them and a scraper that reads four will.

Use a stable identifier as the key whenever the page publishes one. A product ID, an ISBN or a canonical URL path beats a title, because titles are marketing copy and change without the product changing. Published identifier schemes exist precisely so that independent parties can agree on identity — the ISBN URN namespace specification is the book-world example, and a scraped page that exposes one has already solved the matching problem for you.

Stage 3: Block Before Comparing Pairs

Fuzzy comparison is pairwise, and pairwise is quadratic. For 66 entities that is 2,145 comparisons; for 10,000 it is just under 50 million.

Blocking cuts the field by only comparing records that already share something cheap:

blocks = defaultdict(list)
for entity in merged:
    blocks[norm(entity["title"])[:4]].append(entity)

blocked_pairs = sum(len(b) * (len(b) - 1) // 2 for b in blocks.values())

[4] 66 entities
    all-pairs comparisons  2145
    blocked on 4-char key  154 across 48 blocks
    reduction              92.8%

The trade is explicit: a record whose title starts differently is never compared, so a block key that is too aggressive hides real matches. Blocking on the first four characters misses a pair like The Elephant Tree against Elephant Tree because the article moved. Common answers are to block on a sorted-token prefix, on a numeric identifier, or on several keys at once and take the union of the candidate pairs.

Stage 4: Fuzzy Matching, and When It Is Not Needed

Running the fuzzy pass over this catalogue produced a result worth reporting honestly:

[5] fuzzy near-duplicates above 90 (token_sort_ratio)
    brute force 2145 pairs in 2.5 ms -> 0 candidate(s)

Nothing. After normalisation and exact grouping, one clean catalogue had no near-duplicates left. A fuzzy pass here would be code that never fires.

Fuzzy matching earns its place when records arrive from sources that format titles differently. Taking one real title and rendering it the way three different listings would carry it:

from rapidfuzz import fuzz

VARIANTS = [
    "A Study in Scarlet (Sherlock Holmes #1)",
    "A Study In Scarlet - Sherlock Holmes Book 1",
    "A Study in Scarlet, Sherlock Holmes #1 [Paperback]",
]
keys = [norm(v) for v in VARIANTS]
print("distinct exact keys:", len(set(keys)))
for i in range(len(keys)):
    for j in range(i + 1, len(keys)):
        print(f"ratio {fuzz.ratio(keys[i], keys[j]):5.1f} | "
              f"token_sort {fuzz.token_sort_ratio(keys[i], keys[j]):5.1f} | "
              f"token_set {fuzz.token_set_ratio(keys[i], keys[j]):5.1f}")

distinct exact keys: 3
ratio  93.5 | token_sort  93.5 | token_set 100.0
ratio  87.8 | token_sort  87.8 | token_set 100.0
ratio  85.1 | token_sort  82.8 | token_set  93.5

Three keys for one book — the exact-key stage cannot help here. And the scorer changes the answer more than the threshold does. ratio compares the strings as sequences and is dragged down by the [Paperback] suffix; token_set_ratio compares the sets of tokens, so extra words cost nothing and the first two variants score a clean 100.

Reconciling records from several sources? The Scrapeless free plan covers enough requests to collect the second catalogue that makes the duplicates appear.

Choosing the Threshold From Your Own Data

A threshold is only defensible against measured separation. Two numbers bound it here:

Anything between those two separates the sets cleanly on this data. The method generalises: score a sample of known matches and a sample of known non-matches, look at where the distributions stop overlapping, and put the threshold in the gap. A single global number copied from an article is a guess about someone else's data.

Where the distributions do overlap, the honest answer is a review band — auto-merge above the upper bound, auto-reject below the lower, and queue what lands between them. Statistical record linkage has treated the problem this way for decades, and the US Census Bureau's record-linkage research is the standard reference for the probabilistic framing.

Stage 5: Survivorship

Deciding that two records are the same leaves the question of what the merged record says. Discarding the loser silently throws away the evidence that the match happened.

def survivor(group):
    best = sorted(group, key=lambda r: (r["href"] is None, len(r["href"] or "")))[0]
    return {**best,
            "seen_in": sorted({g["source"] for g in group}),
            "prices": sorted({g["price"] for g in group})}

[3] 71 records -> 66 entities
    merged example: 'Sharp Objects' seen_in=['catalogue1', 'mystery'] prices=['£47.82']

Two properties of that merged record matter. seen_in keeps the provenance, so a wrong merge is traceable afterwards instead of invisible. And prices is a set rather than a single value: when two sources disagree, the disagreement is the interesting part, and collapsing it to whichever record happened to sort first destroys it.

Field-level rules beat a whole-record winner. Longest description, most recent timestamp, most complete record, highest-trust source — chosen per field rather than per record — is what keeps a merge from inheriting one source's gaps.

Where This Sits in a Pipeline

Deduplication belongs in the transform step, after extraction and before the write. Running it earlier means normalising strings you have not parsed yet; running it later means the duplicates are already in the table and the fix becomes a migration.

Collecting the same products across several sources is what makes the stage necessary in the first place — the competitive pricing pipeline has exactly that shape, and the Universal Scraping API is what keeps the record shape consistent when one of those sources renders client-side. Pricing lists what the additional sources cost.

Conclusion

Entity resolution is four cheap stages before one expensive one. Normalisation decides what counts as the same string, exact grouping catches everything that agrees — 5 groups and 10 records here — blocking removes 92.8% of the pairs nobody needs to compare, and only what survives that reaches the fuzzy scorer.

Two findings are worth carrying into your own data. The scorer matters more than the threshold: 87.8 against 100.0 on the same pair. And measure the separation before picking a number, because the 63.4-to-93.5 gap that made the choice obvious here is a property of this catalogue, not a constant.

Ready to reconcile records from more than one source? Start with the Scrapeless free plan and collect the second catalogue that makes the duplicates visible.

FAQ

Q: How do I remove duplicates from scraped data?

Normalise the key field, group on it, then merge each group. Case folding, Unicode NFKD normalisation and punctuation stripping turn visually identical strings into one key, and grouping is O(n) rather than pairwise. On the 71 records above that collapsed 10 records into 5 entities without any similarity scoring. Reach for fuzzy matching only for what survives that pass.

Q: What is entity resolution?

Deciding which records refer to the same real-world thing and consolidating them into one canonical record. Deduplication is the same operation within a single source; the term is usually reserved for the harder cross-source case, where no shared identifier exists and the decision has to be made from field similarity.

Q: What is blocking and why does it matter?

Only comparing records that already share a cheap key, so the pairwise stage does not run over everything. 66 entities are 2,145 possible pairs; a four-character prefix key cut that to 154, a 92.8% reduction. The cost is that records whose key differs are never compared, so a block key that is too tight silently hides matches.

Q: Which fuzzy matching scorer should I use?

token_set_ratio for titles that pick up extra words from different sources, since it compares token sets and ignores extras — it scored 100.0 where ratio gave 87.8 on the same pair. Use ratio when position and order carry meaning, such as codes or addresses. Test both against known matches from your own data before choosing.

Q: What similarity threshold should I set?

Measure it rather than copy it. Score a sample of known matches and known non-matches and put the threshold where the distributions stop overlapping. Here the highest score between different books was 63.4 and the lowest among renderings of one book was 93.5, so anything in that band worked. Where the two overlap, auto-merge above, auto-reject below, and queue the middle for review.

Q: Which record should survive a merge?

Choose per field, not per record. Take the longest description, the most recent price, the most complete address, and keep the provenance — the merged entity above retains seen_in and the full set of observed prices. Keeping the source list makes a bad merge traceable; keeping every observed price preserves the disagreement between sources, which is often the signal you wanted.