Build a Google SERP Snapshot Dataset for SEO Research

TL;DR:

• Google SERP tracking needs a record of the search, not only a ranking column. Save the exact request, client timestamps, processing state, and raw response together.
• Build history from your own observations. A current search request does not fill in the days before collection began.
• Compare organic results within a fixed context. Changes in query, country, language, or collection depth create a different comparison group.

A rank value loses its meaning when the spreadsheet no longer records which search produced it. The same page can appear under different queries, in different markets, and beside different result modules. A useful dataset keeps those conditions attached to every observation.

Google SERP tracking starts with repeatable collection and a clear definition of what counts as a comparable result. Scrapeless Google Search API provides structured search data; the application supplies the storage policy and the history. This article builds that boundary explicitly, from a capture file to a dataset another analyst can inspect.

Define the Observation Before the Schedule

A snapshot is the saved outcome of one configured search request. It includes the submitted input and the response received by the client. It does not establish everything every user saw for that query.

Write down the dataset's scope before scheduling collection. Choose a query list, market context, language, result type, and pagination policy. Assign each query to a research topic so that a later report can explain why the query was included. A small reviewed sample is easier to interpret than a large collection whose intent changes every week.

Separate the collection schedule from the search context. The context identifies which observations can be compared; the timestamps identify when the application requested and received them. A client receipt timestamp is not a claim about the exact moment Google generated the page.

Keep Raw Records and Derived Rows

The raw record should retain the request, response, and collection state. A derived result row should point back to that record through a stable run identifier.

Keep fields such as position, title, link, and snippet in the organic-result projection when available. Also keep the array ordinal as a separate field if you need to preserve source order. A missing returned position must remain missing; an array index should not silently replace it.

The JSON data model gives arrays, objects, and null different meanings. Preserve those distinctions in the archive even if the reporting layer later uses a simpler table. The raw response makes it possible to change a mapping without recollecting an observation that can no longer be reproduced.

Store the parser version with derived data. When a parser is corrected, regenerate the affected projection and mark the new version. Treating a parser update as a market change would create a false trend.

Prerequisites for a Capture File

Use Python with the requests package installed and a Scrapeless API key supplied through SCRAPELESS_API_KEY. Install the client with python3 -m pip install requests. The remaining imports use the standard library.

Save the script below as capture_snapshot.py and run python3 capture_snapshot.py. It writes one uniquely named JSON file under snapshots. Ensure that directory is writable and included in your storage policy. The example is a local capture program; it does not supply a scheduler, database, or task-result retrieval service.

An authenticated run requires your own account key. No live account result is claimed here. The request interface has been checked against the current Google Search request workflow; the surrounding file and state handling can be tested locally.

Capture the Request and Its Processing State

The capture program sends actor: scraper.google.search with the search settings inside input. Authentication uses the x-api-token header.

Note: This block requires SCRAPELESS_API_KEY and service access. It has not been executed against a live account for this article. Pending task responses are saved for inspection; task-result retrieval is outside this example.

import json
import os
import uuid
from datetime import datetime, timezone
from pathlib import Path
import requests

def capture(input_parameters, directory="snapshots"):
    key = os.environ["SCRAPELESS_API_KEY"]
    request = {"actor": "scraper.google.search", "input": input_parameters}
    record = {
        "schema_version": 1,
        "run_id": str(uuid.uuid4()),
        "requested_at": datetime.now(timezone.utc).isoformat(),
        "request": request,
    }
    try:
        response = requests.post(
            "https://api.scrapeless.com/api/v1/scraper/request",
            headers={"x-api-token": key}, json=request, timeout=120,
        )
        record["http_status"] = response.status_code
        record["received_at"] = datetime.now(timezone.utc).isoformat()
        try:
            payload = response.json()
        except ValueError:
            payload = None
            record["response_text"] = response.text
        record["response"] = payload
        organic = payload.get("organic_results") if isinstance(payload, dict) else None
        if response.status_code == 201:
            record["state"] = "pending"
        elif response.status_code != 200:
            record["state"] = "http_error"
        elif not isinstance(organic, list) or any(not isinstance(x, dict) for x in organic):
            record["state"] = "unmapped"
        else:
            record["state"] = "observed" if organic else "empty"
    except requests.RequestException as exc:
        record["state"] = "transport_error"
        record["error_type"] = type(exc).__name__
    root = Path(directory)
    root.mkdir(parents=True, exist_ok=True)
    path = root / (record["run_id"] + ".json")
    with path.open("x", encoding="utf-8") as handle:
        json.dump(record, handle, ensure_ascii=False, indent=2)
    print(path, record["state"])
    return path

if __name__ == "__main__":
    capture({"q": "coffee", "gl": "us", "hl": "en", "start": 0,
             "google_domain": "google.com", "device": "desktop"})

The script writes the record after the HTTP operation, including error outcomes that are useful for collection coverage. Its exception record contains the exception type rather than a full diagnostic string that might expose unnecessary request details. Store keys separately from archived search data.

The 120 timeout is an application choice. It is not a service response-time guarantee. Python's timezone-aware timestamps make the client times explicit; use the same convention when another collector writes into the dataset.

Start Scraping with Scrapeless

Power up your web scraping and automation workflow with Scrapeless!
Sign up today and get $5 in free credit — no credit card required.

Claim your free credit now in the Scrapeless Dashboard.

Treat Missing Collection as Its Own Result

A failed observation does not mean a tracked domain disappeared. The state field protects the report from that mistake.

observed means a successful data response contained a nonempty organic array of objects. empty means the array was present and empty. unmapped means the successful response did not match that minimal shape. These are application labels, not additional API status codes.

pending records the documented HTTP 201 task state. Keep the returned task identifier with the raw response and use a separately verified result-retrieval workflow before counting that run as observed. HTTP errors and transport errors belong in the collection report, not in a ranking movement chart.

When preparing a weekly view, show how many planned observations were usable. Keep a list of missing or unresolved runs next to the result comparison. A chart that quietly excludes failed collection can look stable while its evidence becomes thinner.

Compare Like-for-Like Organic Results

A comparison key should include every submitted search setting that can affect the observation. The simplest conservative implementation serializes the whole request object with sorted keys, excluding only application timestamps and run identifiers.

Python's deterministic JSON serialization options support sorted dictionary keys. This gives your application a repeatable representation of the submitted settings. It does not prove that different settings are semantically equivalent, and it does not freeze upstream search behavior.

Keep start in the key when comparing page slices. If the report combines multiple pages into one collection window, define that higher-level window separately and mark missing pages. Do not mix a first-page observation with a deeper collection and call the difference a ranking gain.

Use the returned organic positions only within the interpretation supported by your collected response. Keep image, local, and other modules separate. Search Console's position measurement rules describe a different reporting system; a sampled API position should not be relabeled as Search Console average position.

Build a Reviewable Change Log

A useful change record identifies the old run, new run, context, affected URL, and the rule that detected the change. It should describe an observation before suggesting a cause.

For a domain, distinguish “present in both captured slices,” “newly observed in this slice,” and “not observed in the later slice.” The last label is narrower than “removed from Google.” The domain may be outside the collected depth, and a URL replacement can leave domain presence unchanged.

For a URL, retain both exact-link and normalized-host comparisons. Normalization can help group pages, but dropping paths, parameters, or subdomains can also merge things the research cares about. Document each normalization rule and keep the original link beside the derived value.

Route changes to human review with the saved evidence. Page updates, query context, and broader search changes may all deserve investigation. A pair of snapshots alone cannot establish which one caused the movement.

Conclusion

Start the dataset with an explicit scope and a raw capture record. Add derived organic rows only after the run state is understood, then compare records whose request settings match. The result is a history your team can audit, with collection gaps visible rather than converted into ranking claims.

A Python search collection walkthrough gives additional background; use the current request interface shown here for this dataset.

Build Your Next Search Observation

Configure Google Search API around the questions your team needs to answer. Check Scrapeless pricing before setting collection frequency. The Google Search parameter model explains the context controls used in this workflow.

Discuss your implementation with the community on Discord or Telegram.

FAQ

Q: Does this retrieve historical Google rankings from before collection began?

No. This workflow builds history from observations you save. It does not create earlier snapshots or provide a historical ranking database.

Q: Is an empty organic array the same as a failed request?

No. A present empty array is recorded as empty; HTTP failures, transport failures, pending tasks, and an unmapped response have separate states.

Q: Can different countries share the same ranking history?

They can share a storage system, but should remain separate comparison groups. Country is part of the request context.

Q: Does a result position measure visits or revenue?

No. It describes the returned search observation. Traffic and business outcomes need their own evidence and matching definitions.

Q: How frequently should snapshots be collected?

Choose a cadence that matches the research question, budget, and review capacity. Record missed observations and avoid implying that a sampled schedule captures every change.