Paginate Google Search Results with Traceable Request Context

TL;DR:

• Google search API pagination changes the collected slice. Keep start and the complete request with every response rather than treating pages as interchangeable batches.
• Continue only within a defined plan and a reviewed response contract. The example inspects the returned next link before changing the offset and stops on ambiguous continuation data.
• Keep pending work and duplicate observations visible. Store each raw page response; deduplication belongs to a derived view, and HTTP 201 is not an empty page.

Pagination can hide a change in the experiment. A collector may advance the offset while losing a country setting, or combine a pending response with completed pages and call the result complete. The resulting URL list does not reveal how those differences entered the collection.

Scrapeless Google Search API supports search offsets through start. This Google search API pagination guide builds a deliberately bounded collector with a record for every request and a reason for stopping. It demonstrates application control flow without promising access to every result or an uninterrupted historical ranking sequence.

Prerequisites and the Bounded Collection Plan

The program uses Python's standard library and requires an account API key in SCRAPELESS_API_KEY for real collection. The environment-variable name is a convention of this example. Keep the key outside request-body logs and generated capture files.

The Google Search parameters describe start as the result offset, with examples of 0, 10, and 20. The program limits its plan to those offsets. This is an application bound, not a statement about the service's maximum depth or a guarantee that every page contains a fixed number of organic items.

The base query is coffee with gl=us, hl=en, and desktop input. These settings are an illustrative request configuration. They are not presented as an observed search sample. Change the configuration only after deciding the scope of your own collection.

Authenticated requests were not executed for this article because no account key was supplied. Local tests exercise the continuation and state logic with synthetic responses. A live account run remains a prerequisite for validating current response behavior.

Keep a Page Offset Beside Every Raw Capture

The Google Search request workflow submits scraper.google.search to POST https://api.scrapeless.com/api/v1/scraper/request with authentication in x-api-token. The collector records the submitted body separately from the response and adds client timestamps and its own run identifier.

HTTP 200 contains task data; HTTP 201 represents a task still being processed. The latter response must be saved before stopping. Its task identifier remains in the raw response for a separately verified completion workflow; this example does not invent a task-retrieval endpoint.

The capture envelope is application-owned. Its request, response, timestamps, and error field are not asserted as the native API wrapper. Keeping them separate follows the JSON data model and preserves the underlying evidence for a later mapper.

Inspect Continuation Evidence Before Advancing

The quickstart response example includes pagination.next, but the sample URL is abbreviated. Do not execute that abbreviated value. The code requires an actual, nontruncated HTTPS Google search URL before using its offset as a continuation hint.

The example accepts only reviewed Google hosts and checks that query, country, and language in the returned link match the base configuration. It preserves all submitted base settings and changes only start. A missing country field in the next link causes a review stop rather than an inference about what the service intended.

These checks use URL component parsing. They define the application's narrow continuation policy, not every valid Google URL. The collector never follows the returned URL directly; it sends the next request to the same API endpoint with the reviewed parameter-mode body.

Run the Collector With Explicit Stop Reasons

Save the program as paginate_search.py. With your key set in the environment, run python3 paginate_search.py in a writable directory. The script creates a uniquely named output directory, sends requests sequentially, and keeps each page capture plus collection-summary.json.

Note: Live API collection is a prerequisite and was not performed for this article. The code's local control flow was tested using synthetic responses. Inspect real account output before relying on the next-link policy, and verify task retrieval separately if a request returns HTTP 201.

import json
import os
import uuid
from datetime import datetime, timezone
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.parse import parse_qs, urlsplit
from urllib.request import Request, urlopen

ENDPOINT = 'https://api.scrapeless.com/api/v1/scraper/request'
OFFSETS = (0, 10, 20)
BASE = {'q': 'coffee', 'gl': 'us', 'hl': 'en', 'device': 'desktop'}


def now():
    return datetime.now(timezone.utc).isoformat()


def fetch(body, key):
    request = Request(ENDPOINT, data=json.dumps(body).encode(), method='POST',
                      headers={'Content-Type': 'application/json', 'x-api-token': key})
    try:
        with urlopen(request, timeout=60) as response:
            status, raw = response.status, response.read().decode('utf-8')
    except HTTPError as error:
        status, raw = error.code, error.read().decode('utf-8', errors='replace')
    except (URLError, TimeoutError) as error:
        return None, None, type(error).__name__
    try:
        return status, json.loads(raw), None
    except json.JSONDecodeError:
        return status, {'unparsed_body': raw}, 'response_not_json'


def next_offset(payload, current):
    pagination = payload.get('pagination')
    link = pagination.get('next') if isinstance(pagination, dict) else None
    if not isinstance(link, str) or not link:
        return None, 'next_link_unavailable'
    try:
        parts = urlsplit(link)
        if (parts.scheme != 'https' or parts.hostname not in ('google.com', 'www.google.com')
                or parts.username or parts.password or parts.port or parts.path != '/search'
                or parts.fragment or '...' in link or '…' in link):
            return None, 'next_link_needs_review'
        query = parse_qs(parts.query, keep_blank_values=True)
        if any(query.get(k) != [BASE[k]] for k in ('q', 'gl', 'hl')):
            return None, 'next_context_needs_review'
        values = query.get('start', [])
        if len(values) != 1 or not values[0].isascii() or not values[0].isdigit():
            return None, 'next_offset_needs_review'
        offset = int(values[0])
        if offset <= current or offset not in OFFSETS:
            return None, 'next_offset_outside_plan'
        return offset, None
    except ValueError:
        return None, 'next_link_needs_review'


def collect(directory, key):
    directory = Path(directory)
    directory.mkdir(parents=True, exist_ok=False)
    seen, visited, pages = set(), set(), []
    offset, stop = 0, None
    while offset in OFFSETS and offset not in visited:
        visited.add(offset)
        body = {'actor': 'scraper.google.search', 'input': dict(BASE, start=offset)}
        started = now()
        status, payload, error = fetch(body, key)
        run_id = uuid.uuid4().hex
        capture = {'run_id': run_id, 'requested_at': started, 'received_at': now(),
                   'request': body, 'http_status': status, 'response': payload, 'error': error}
        filename = f'{offset}-{run_id}.json'
        (directory / filename).write_text(json.dumps(capture, ensure_ascii=False, indent=2), encoding='utf-8')
        page = {'start': offset, 'capture': filename, 'new_exact_urls': None}
        pages.append(page)
        if status == 201:
            stop = 'pending'
            break
        if status != 200 or error:
            stop = 'collection_error'
            break
        rows = payload.get('organic_results') if isinstance(payload, dict) else None
        if not isinstance(rows, list) or any(not isinstance(row, dict) for row in rows):
            stop = 'unmapped_organic_results'
            break
        links = {row['link'] for row in rows if isinstance(row.get('link'), str) and row['link']}
        page['new_exact_urls'] = len(links - seen)
        seen.update(links)
        if not rows:
            stop = 'empty_organic_slice'
            break
        if not links:
            stop = 'no_usable_link_strings'
            break
        if page['new_exact_urls'] == 0:
            stop = 'no_new_exact_urls'
            break
        if len(visited) == len(OFFSETS):
            stop = 'planned_page_limit'
            break
        offset, stop = next_offset(payload, offset)
        if stop:
            break
    summary = {'pages': pages, 'stop_reason': stop, 'unique_exact_url_strings': len(seen)}
    (directory / 'collection-summary.json').write_text(json.dumps(summary, indent=2), encoding='utf-8')
    return summary


if __name__ == '__main__':
    key = os.environ['SCRAPELESS_API_KEY']
    output = 'search-pages-' + uuid.uuid4().hex
    print(json.dumps(collect(output, key), indent=2))

The timeout is a local client setting, not a service performance claim. A transport exception or non-JSON response produces an error record and stops the collection. The program keeps earlier captures instead of presenting a partial run as a complete sweep.

Start Scraping with Scrapeless

Power up your web scraping and automation workflow with Scrapeless!
Sign up today and get $5 in free credit — no credit card required.

Claim your free credit now in the Scrapeless Dashboard.

Separate Deduplication From the Search Evidence

The summary counts new exact URL strings on each page while retaining every original response. Repeated URLs remain in those captures with their page context. This allows a reviewer to inspect overlap rather than losing it during export.

Exact-string deduplication is intentionally narrow. It does not merge tracking variants, fragments, canonical URLs, or different pages from one domain. A broader grouping policy belongs in a separately versioned transform and should retain the original links.

The program stops when a usable page contributes no new exact URL strings. That is a collection-budget decision, not proof that no further relevant pages exist. Likewise, a present empty organic array stops this run without establishing an exhaustive search boundary.

Interpret a Partial Collection Honestly

Read the summary's stop reason before using its URL count. planned_page_limit means the local bound was reached. next_link_unavailable means continuation was not established. Pending, unmapped, and collection-error states describe unfinished or unusable evidence, not successful end-of-results detection.

Each request has its own timestamp. Sequential pages were not collected simultaneously, and the program does not claim a shared server session across calls. Preserve the collection window when comparing this run with another one.

The provenance model helps separate raw page observations, the collection activity, and a derived deduplicated list. Even a small filesystem layout can preserve those relationships if the summary points back to every capture.

Conclusion

Plan a limited collection, preserve every submitted request, and advance only when the response supports the application's continuation rule. Explicit stop reasons and retained duplicates make a partial dataset understandable without claiming it contains every Google result.

A SERP snapshot dataset approach can help organize the saved observations after collection; keep page offsets visible when deciding which records are comparable.

Build Your Next Search Observation

Use Scrapeless Google Search API for the search data in this workflow. Review Scrapeless pricing when planning collection, and keep the Google Search parameters beside your configuration.

Discuss your implementation with the community on Discord or Telegram.

FAQ

Q: Does start=10 guarantee ten organic rows?

No. start specifies an offset. Inspect the actual organic array instead of deriving its length from the requested offset.

Q: Does this script collect every Google result?

No. It is bounded by a small application plan and stops when continuation or usable data is unavailable.

Q: What happens to HTTP 201?

The response is saved and collection stops as pending. A verified task-completion workflow is required before interpreting the final search data.

Q: Why stop when a next link omits a context field?

The example requires explicit query, country, and language agreement. Missing evidence triggers review rather than a silent assumption about continuation.

Q: Are repeated URLs removed from the raw captures?

No. Only the summary's exact-string count is deduplicated. Original observations remain available for inspection.