How to Build an Incremental Web Scraping Pipeline With Scrapeless

TL;DR:

• A conditional request turns an unchanged page into a 304 with a 0-byte body — the full response was 50,388 bytes, and ten conditional checks ran in 4.21 s against 5.52 s for ten full fetches.
• Content hashing detects change but saves no bandwidth: the second fetch of a validator-less page still transferred 11,064 bytes before the hash could be compared.
• Per-record fingerprints are what shrink the write. One moved price took a 20-record page down to 1 row written downstream.
• Fingerprint the fields you care about rather than the whole record, or an unrelated field changing marks everything dirty.
• HTTP validators describe the document the server sent, so they stop being useful the moment the records arrive through client-side rendering.
• A 304 is only available if the previous run stored the validator, which makes the state table part of the pipeline rather than an optimisation.
• Put an incremental crawl on rendered pages with the Scrapeless free plan.

Most scheduled scrapers re-download everything, re-parse everything, and rewrite every row, then report success. That works until the catalogue grows, at which point the daily job is spending almost all of its time confirming that yesterday's data is still yesterday's data.

Incremental scraping is the opposite arrangement: ask what changed, and do work only where the answer is yes. The question can be asked at three levels, and each one saves something different — bandwidth, parsing, or writes. Each layer below was measured against the same live catalogue page, so the trade-offs come with numbers rather than adjectives.

Pipeline at a Glance

The flow is check → fetch only if changed → extract → compare fingerprints → write only the differences. The layers stack: the HTTP check is cheapest and least often available, the record check is always available and costs a full fetch.

Checking less often is the other half of the same problem. The Robots Exclusion Protocol is where a site states what it wants crawled, and an incremental design is what lets a scraper honour a slower cadence without falling behind — each check costs a round trip instead of a full transfer.

Layer 1: Let the Server Answer

An HTTP validator is the server's own opinion about whether its representation changed. Two exist: ETag, an opaque version token, and Last-Modified, a timestamp. The first response carries them.

import requests

URL = "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
first = requests.get(URL, timeout=30)
first.raise_for_status()
etag = first.headers["ETag"]
print(f"first  GET  {first.status_code}  {len(first.content)} bytes  ETag={etag}")

second = requests.get(URL, headers={"If-None-Match": etag}, timeout=30)
print(f"second GET  {second.status_code}  {len(second.content)} bytes")
print(f"bytes avoided: {len(first.content) - len(second.content)}")

first  GET  200  50388 bytes  ETag=W/"63e40de8-c4d4"
second GET  304  0 bytes
bytes avoided: 50388

The 304 carries no body at all. The conditional-request mechanism is defined so the client keeps the copy it already has, which means the scraper has to have kept one.

Last-Modified works the same way through a different header:

third = requests.get(URL, headers={"If-Modified-Since": first.headers["Last-Modified"]}, timeout=30)
print(first.headers["Last-Modified"], "->", third.status_code, len(third.content), "bytes")

Wed, 08 Feb 2023 21:02:32 GMT -> 304 0 bytes

Prefer the ETag when both are present. The HTTP semantics specification makes Last-Modified a one-second-resolution timestamp, so two changes inside the same second are indistinguishable, while an entity tag is free to change on any edit.

The saving is real but it is not the whole run:

10 conditional GETs   4.21s
10 full GETs          5.52s

Conditional checking ran 1.31 times faster and left 492 KB untransferred. The request still costs a round trip — what disappears is the body, not the connection.

Layer 2: Hash the Document When the Server Says Nothing

Plenty of targets send neither validator. Then the only way to know whether the document changed is to fetch it and look. A cryptographic digest is the standard tool for that comparison — the SHA-256 standard gives a fixed-width value where any single-byte difference produces an unrelated digest, so equality is a reliable "nothing moved" signal.

import hashlib
import requests

q1 = requests.get("https://quotes.toscrape.com/", timeout=30)
q1.raise_for_status()
print("ETag:", q1.headers.get("ETag"), " Last-Modified:", q1.headers.get("Last-Modified"))

q2 = requests.get("https://quotes.toscrape.com/", timeout=30)
h1 = hashlib.sha256(q1.content).hexdigest()
h2 = hashlib.sha256(q2.content).hexdigest()
print(f"sha256 run 1: {h1[:16]}...")
print(f"sha256 run 2: {h2[:16]}...")
print(f"unchanged: {h1 == h2}   bytes still transferred: {len(q2.content)}")

ETag: None  Last-Modified: None
sha256 run 1: efdc2605a2062dce...
sha256 run 2: efdc2605a2062dce...
unchanged: True   bytes still transferred: 11064

Note what this layer does and does not buy. The hash correctly reports no change, and 11,064 bytes crossed the network anyway. Document hashing saves parsing, database writes, downstream alerting and any re-embedding — never bandwidth.

It also has a false-positive problem that the numbers here do not show. Pages carrying a session token, a rotating advertisement or a rendered timestamp produce a different hash on every fetch while the data is identical. Hashing the extracted records instead of the raw body is what makes the signal stable, which is the next layer.

Layer 3: Fingerprint the Records

The layers above answer questions about a document. What a pipeline usually needs to know is which rows to write.

import json, hashlib

def fingerprint(record):
    payload = json.dumps({k: record[k] for k in ("price", "rating")}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:16]

Fingerprinting a chosen subset of fields rather than the whole record is the decision that makes this work. Include a field that changes on its own — a view count, a "last seen" timestamp, a position in a ranked list — and every record looks dirty on every run.

Store one fingerprint per record, then compare the next scrape against it:

known = {title: (fp, price) for title, fp, price in conn.execute("SELECT title, fp, price FROM snap")}
new, changed, same = [], [], 0
for record in second_pass:
    previous = known.get(record["title"])
    if previous is None:
        new.append(record)
    elif previous[0] != fingerprint(record):
        changed.append((record["title"], previous[1], record["price"]))
    else:
        same += 1

Against the live 20-record page with one price altered to stand in for a real move:

parsed 20 records from the live page
stored baseline: 20 fingerprints
example fingerprint: 'Sharp Objects' -> e974692e9d935048

unchanged 19 | changed 1 | new 0
  CHANGED A Murder in Time                     £16.64 -> £99.99

rows written downstream: 1 of 20

One row written instead of twenty. On a catalogue where the daily reality is that almost nothing moves, that ratio is the entire argument for fingerprinting: the write volume tracks the change rate rather than the catalogue size.

Keeping the previous value alongside the fingerprint is what turns detection into a usable event. £16.64 -> £99.99 is a price-change record; a dirty flag is only a hint that something happened.

Running an incremental crawl against pages that render client-side? The Scrapeless free plan covers enough requests to build the baseline and the first few diffs.

Where the HTTP Layer Stops Applying

A validator describes the document the server sent. When the records arrive through client-side rendering, that document is the application shell, and its ETag tracks the shell's deployment rather than the catalogue.

The consequence is specific: a page can return 304 while the prices behind it have all moved, because the shell genuinely did not change. Any target whose content is assembled in the browser has to be checked at the record layer, using the Universal Scraping API to render before comparing.

The same applies to an internal JSON endpoint the page calls. Those often do send validators, and when they do, the top layer works again against the payload that actually carries the records.

Storing the State

None of this works without somewhere to keep what the last run learned. The state a pipeline needs is small:

Per record, the table holds the key, the fingerprint, and whichever previous values you want to report on. Both fit alongside the scraped data in the same database, and the ETL pipeline shape is unchanged — a check step is added in front of the extract.

One operational note that is easy to miss: an ETag is only valid for the URL that issued it. Replaying a stored tag against a different query string or a paginated variant will produce a 200 and a full body, which is correct behaviour rather than a fault.

Conclusion

Incremental scraping is three questions, and knowing which one you are asking decides what you save. The HTTP validator saves the body — 50,388 bytes became a 0-byte 304. The document hash saves parsing and writes but never bandwidth, since the 11,064 bytes arrive before the comparison. The record fingerprint saves the write, and took a twenty-record page to a single row.

Start with the validator when the server offers one, fall back to hashing the extracted records rather than the raw body, and fingerprint only the fields whose movement you actually care about. Pricing lists what the remaining fetches cost once the unchanged ones stop happening.

Ready to stop re-scraping pages that have not moved? Start with the Scrapeless free plan and build the baseline your next run compares against.

FAQ

Q: What is incremental web scraping?

Scraping only what changed since the last run, rather than re-collecting the whole target. It is implemented as a check that runs before the fetch or before the write: a conditional HTTP request, a document hash comparison, or a per-record fingerprint comparison. The measured effect here was 50,388 bytes avoided at the HTTP layer and 19 of 20 rows skipped at the record layer.

Q: How do I use ETag in a scraper?

Store the ETag header from the response, then send it back as If-None-Match on the next request for that same URL. A server that agrees nothing changed replies 304 with an empty body, which is the signal to skip the rest of the pipeline. The tag is tied to the exact URL, so a stored tag replayed against a different query string returns a normal 200.

Q: What if a site sends no ETag or Last-Modified?

Fetch and compare hashes instead. Hash the extracted records rather than the raw HTML — a raw-body hash changes when a session token, advertisement or rendered timestamp changes, which marks the page dirty while the data is identical. This costs the bandwidth either way; the saving is in parsing, writing and anything downstream.

Q: Should I hash the whole record or specific fields?

Specific fields. A whole-record hash includes anything the page happens to carry, so a rank position or a "last viewed" counter makes every record look changed on every run. Fingerprinting only the fields whose movement matters is what produced the stable 19-unchanged result above.

Q: Can a page return 304 while its data has actually changed?

Yes, on client-rendered pages. The validator describes the HTML document the server sent, which for a single-page application is the shell rather than the records, so the tag tracks deployments of the shell. Those targets need comparison at the record layer after rendering, or against the internal JSON endpoint that carries the data.

Q: How much does incremental scraping actually save?

It depends on which layer answers. In the measurement above, conditional requests removed the response body entirely and ran 1.31 times faster over ten checks, and the record layer cut writes by 95% on a page where one of twenty items moved. The round trip itself remains in every case, so the saving scales with page size and change rate rather than with request count.