How to Scrape Google AI Overviews: A Complete Guide for SEO and Brand AI Visibility Monitoring

Key Takeaways:

• One actor, the whole AI Overview surface. scraper.overview returns the AI Overview body (markdown + plain text), the cited sources panel, the related-search web sources, sponsored placements, and the shopping-flag set — all as structured JSON from a single HTTP POST.
• Country-pinned residential egress. The input.country field routes the request through a geo-matched residential proxy so the AI Overview Google generates is the one a real user in that country would see. Verified end-to-end against US and GB.
• One canonical envelope. Every successful response is { status, task_id, task_result }. task_result.content is markdown with [N] citation refs; task_result.rawtext is the same body without citations; task_result.source and task_result.web_source are the two ranked link panels.
• Pair with scraper.google.search and scraper.aimode. The AI Overview is one Google AI surface — the AI Mode tab and the classic organic SERP are siblings. Same Scrapeless account, same auth header.
• Free to start. New Scrapeless accounts include free Scraper API credits — sign up at Scrapeless Webiste.

Introduction: structured access to Google's AI Overview

Google's AI Overview (AIO) sits at the top of the SERP for an increasing share of informational, comparison, and shopping queries. For SEO teams it has displaced the Featured Snippet as the load-bearing answer surface; for brand monitoring it is now the first thing a prospect reads about a product; for AI-grounding pipelines it is a curated set of citations Google has already vetted for relevance.

Scraping the AIO directly from the rendered SERP is a moving target. The block lazy-loads behind a "generating" placeholder, the markup rotates across A/B variants, and the citation panel is hydrated independently from the body. A DOM-scraping pipeline can be made to work but it carries the selector-maintenance tax of any front-end target — and it still needs residential egress, JavaScript rendering, and CAPTCHA handling underneath.

The Scrapeless Scraper AI Overview API (actor: "scraper.overview") reduces all of that to one HTTP POST. The caller sends a prompt and a country; the API returns a structured JSON envelope with the AIO body in two formats (markdown with citations, plain text without), the cited sources, the related-search web sources, sponsored placements above the AIO, and the shopping/purchase-link flags. Authentication, proxy routing, JavaScript rendering, lazy-load polling, and selector maintenance are all server-side concerns.

This guide walks through the full integration: why teams use the API, the request and response shape, parameter and field reference, runnable Python and Node.js clients, the error matrix observed in verification, and a short tour of the companion actors (scraper.google.search, scraper.aimode) that round out a production Google-AI pipeline.

What You Can Do With It

• AI Overview rank and presence monitoring. Track which of your target keywords actually surface an AIO, and how often — the AIO trigger rate is itself a high-signal SEO KPI in 2026.
• GEO citation tracking. Pull the list of cited domains for each AIO and aggregate share-of-citation per brand, per topic cluster, per geography. This is the GEO equivalent of organic share-of-voice.
• Brand monitoring on AI answers. Watch which third-party reviews, comparisons, and editorial pieces Google's AIO surfaces when prospects search for your brand or your competitors.
• Competitor visibility audits. Diff the cited-source lists for a competitor's branded queries against your own — the gap is your GEO content roadmap.
• Training data for LLM eval and RAG. Each AIO is a curated query → grounded-answer → citation-set triple. Captured at a fixed timestamp and proxy geography, it is reproducible ground truth for RAG evals and answer-quality benchmarks.
• Sponsored-placement intelligence. The same response carries the ads that ran above the AIO for the same query (advertiser, title, display URL, tracking token). Pair it with the AIO body to study advertiser behavior on AI-augmented SERPs.
• Shopping intent surfacing. task_result.is_overview_shopping, is_shopping, and purchase_link are AIO-level flags that mark queries Google has classified as commercial. Useful for retail-intel pipelines that filter for purchase-intent terms.
• Multi-locale capture. Set input.country per request to read the AIO that real users see in the US, GB, DE, FR, JP, and the rest of the supported residential-proxy footprint.

At Scrapeless, we only access publicly available data while strictly complying with applicable laws, regulations, and website privacy policies. The content in this post is for demonstration purposes only.

Why the Scrapeless Scraper AI Overview API

The API turns Google's AI Overview block — a lazy-loaded, rotating, geo-sensitive surface — into a single structured-JSON HTTP call.

• No DOM scraping, no Playwright, no selector maintenance. Send {prompt, country}, receive {ads, content, rawtext, source, web_source, ...}. The body is delivered as markdown with inline [N] citation refs and as a citation-stripped plain-text twin.
• Country-pinned residential egress. The input.country field routes the request through a geo-matched residential proxy. The AI Overview returned is the one a real user in that country would see — not a generic US fallback.
• Lazy-load handled server-side. AIOs render behind a "generating" placeholder that the actor polls server-side; this polling dominates the ~12–18 s end-to-end latency observed in verification. The caller receives only the settled body and does not implement a wait loop.
• Pre-parsed citation panels. task_result.source is the AI Overview's cited-sources panel (the links Google attributes the answer to); task_result.web_source is the related-search panel that appears under the AIO. Both arrive as arrays of {title, url, snippet, website_name, favicon, thumbnail}.
• Shopping signals as first-class fields. is_overview_shopping, is_shopping, and purchase_link mark commercial-intent AIOs without the caller having to parse the body.
• Same Scrapeless auth as the rest of the product line. The Scraper AI Overview API uses the same x-api-token header as scraper.google.search, scraper.amazon, and the Universal Scraping API. One account, many surfaces.

Get your API key on the free plan at Scrapeless Webiste. The Scraper API line sits alongside the Scraping Browser, Universal Scraping API, and AI Agent in the pricing catalogue.

Prerequisites

• A Scrapeless account and API key — sign up at Scrapeless.
• A terminal with curl (for the smoke test) or Python 3.10+ / Node.js 18+ for the integrated clients below.
• Basic familiarity with HTTP and JSON.

No browser, no Playwright, no proxy plan to buy separately. The actor handles residential egress, JavaScript rendering, anti-detection, and lazy-load polling server-side.

How the Scraper AI Overview API works

The integration is one HTTP POST. Get an API token from the Scrapeless dashboard and store it as an environment variable:

export SCRAPELESS_API_TOKEN=your_token_here

The endpoint is POST https://api.scrapeless.com/api/v2/scraper/execute with header x-api-token: <YOUR_TOKEN> and a JSON body that names the actor and its input:

curl --location 'https://api.scrapeless.com/api/v2/scraper/execute' \
  --header 'Content-Type: application/json' \
  --header "x-api-token: ${SCRAPELESS_API_TOKEN}" \
  --data '{
    "actor": "scraper.overview",
    "input": {
        "prompt":  "best running shoes",
        "country": "US"
    }
}'

A successful call returns HTTP 200 in ~15 seconds end to end and the body is the canonical envelope below.

Request parameters

Response envelope

A successful response is a JSON object with three top-level keys:

{
  "status":    "success",
  "task_id":   "ca132d3f-dc04-464e-b652-53231a8aeb8f",
  "task_result": {
    "ads":                  [ /* sponsored placements above the AIO */ ],
    "content":              "**GraphQL**is an open-source query language ... ([GraphQL][1]) ([Postman Blog][2]) ...\n\n[1]: https://graphql.org/ \"GraphQL — ...\"",
    "rawtext":              "GraphQL is an open-source query language ...",
    "is_overview_shopping": true,
    "is_shopping":          true,
    "purchase_link":        true,
    "metadata":             { "rawUrl": "https://www.google.com/search?ApiType=overview&IsShopping=true&oq=...&q=..." },
    "products":             null,
    "source":               [ /* AI Overview cited sources */ ],
    "web_source":           [ /* related-search web sources below the AIO */ ]
  }
}

task_result field reference:

The envelope also carries status ("success" or "failed") and task_id (a UUID for the execution) at the top level. Store the task_id in your own logs — it is the correlation key when filing a support ticket about a specific execution.

Integrating the API in Python

The full integration is a single POST and a .json() parse. The script below reads the API token from the environment, calls the actor with prompt="best running shoes" and country="US", and prints the AIO body, the top five cited sources, and the count of sponsored placements above the AIO.

import os
import requests

URL = "https://api.scrapeless.com/api/v2/scraper/execute"
HEADERS = {
    "x-api-token":  os.environ["SCRAPELESS_API_TOKEN"],
    "Content-Type": "application/json",
}
BODY = {
    "actor": "scraper.overview",
    "input": {
        "prompt":  "best running shoes",
        "country": "US",
    },
}

resp = requests.post(URL, headers=HEADERS, json=BODY, timeout=60)
resp.raise_for_status()
data = resp.json()

if data.get("status") != "success":
    raise SystemExit(f"AIO scrape failed: {data}")
# If it is "pending" or "running", use the "task_id" to retry and obtain the result data.

result = data["task_result"]

print(f"task_id = {data['task_id']}")
print(f"shopping flags: is_shopping={result['is_shopping']} "
      f"is_overview_shopping={result['is_overview_shopping']} "
      f"purchase_link={result['purchase_link']}\n")

# Plain-text body — best for downstream LLM / embedding pipelines
print("=== AI Overview (rawtext) ===")
print(result["rawtext"][:1200], "...\n")

# Top-5 cited sources from the AI Overview panel
print("=== Top cited sources ===")
for s in result["source"][:5]:
    print(f"  - {s['website_name']:<20}  {s['url']}")

print(f"\nads above AIO: {len(result['ads'])}")

A representative run of this script returns the AIO body in 12–18 seconds and prints six to twelve cited sources for the query.

Bulk capture with retry

In production, wrap the call in a small retry loop. The actor occasionally returns HTTP 400 with {"message":"execution failed","status":"failed","task_id":"..."} when Google does not surface an AIO for the query in a given proxy geography or when the upstream render is momentary. Retrying with a short back-off usually clears it:

import os, time, requests

URL = "https://api.scrapeless.com/api/v2/scraper/execute"
HEADERS = {
    "x-api-token":  os.environ["SCRAPELESS_API_TOKEN"],
    "Content-Type": "application/json",
}

def fetch_aio(prompt: str, country: str = "US", retries: int = 3, backoff: float = 3.0):
    body = {"actor": "scraper.overview", "input": {"prompt": prompt, "country": country}}
    last = None
    for attempt in range(retries):
        resp = requests.post(URL, headers=HEADERS, json=body, timeout=60)
        last = resp
        if resp.status_code == 200:
            payload = resp.json()
            if payload.get("status") == "success":
                return payload["task_result"]
        # transient: 400 with "execution failed" is the retryable case
        if resp.status_code == 400 and "execution failed" in resp.text:
            time.sleep(backoff * (attempt + 1))
            continue
        # hard error: bad auth, bad actor, missing prompt — do not retry
        resp.raise_for_status()
    raise RuntimeError(f"AIO scrape exhausted retries for {prompt!r}: {last.text}")

keywords = [
    "best running shoes",
    "best wireless earbuds for running",
    "how to choose running shoes",
]
for kw in keywords:
    result = fetch_aio(kw, country="US")
    print(f"\n=== {kw} ===")
    print(f"  cited sources: {len(result['source'])}  "
          f"web sources: {len(result['web_source'])}  "
          f"ads: {len(result['ads'])}")

This is the load-bearing pattern for keyword-batch monitoring. Keep concurrency modest — three to five parallel workers per token is a safe starting point — and back off on the execution failed signal rather than treating it as a hard failure.

Get your API key on the free plan: Scrapeless Webiste

Integrating the API in Node.js (18+)

The same call in Node, using the built-in fetch:

const URL = "https://api.scrapeless.com/api/v2/scraper/execute";

async function fetchAIO(prompt, country = "US") {
  const resp = await fetch(URL, {
    method: "POST",
    headers: {
      "x-api-token":  process.env.SCRAPELESS_API_TOKEN,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      actor: "scraper.overview",
      input: { prompt, country },
    }),
  });

  if (!resp.ok) {
    const text = await resp.text();
    throw new Error(`HTTP ${resp.status}: ${text}`);
  }
  const payload = await resp.json();
  if (payload.status !== "success") {
    throw new Error(`AIO scrape failed: ${JSON.stringify(payload)}`);
  }
  return payload.task_result;
}

const result = await fetchAIO("best running shoes", "US");

console.log(`shopping flags: is_shopping=${result.is_shopping} ` +
            `is_overview_shopping=${result.is_overview_shopping}`);

console.log("\n=== AI Overview (rawtext) ===");
console.log(result.rawtext.slice(0, 1200), "...\n");

console.log("=== Top cited sources ===");
for (const s of result.source.slice(0, 5)) {
  console.log(`  - ${s.website_name.padEnd(20)} ${s.url}`);
}

console.log(`\nads above AIO: ${result.ads.length}`);

The retry pattern mirrors the Python version: only retry on HTTP 400 + "execution failed"; treat 401, 4xx with code: 14002 (invalid actor), and the validation error on Prompt as hard failures that no amount of retrying will fix.

Companion actors for end-to-end Google-AI scraping

The Scraper AI Overview API covers one Google AI surface. Production pipelines that monitor brand visibility, build GEO datasets, or train answer-quality models usually want two more.

scraper.google.search — the classic organic SERP

For the ten blue links beneath the AIO, the People Also Ask pairs, the Knowledge Panel, the Featured Snippet, and the Related Searches block, scraper.google.search is the structured counterpart. The cited-source domains in scraper.overview's source array are useful in isolation, but they make far more sense when joined against the organic top-10 for the same query — that join tells you whether a cited domain is also ranking organically, and at what position.

scraper.aimode — the AI Mode tab

Google's AI Mode is a separate, full-page conversational experience that surfaces from the SERP. It rewrites the answer in a more chat-like format, takes follow-up prompts, and uses a different citation panel layout. scraper.aimode is the dedicated actor for it. For a complete picture of how Google's AI Search is presenting a topic, capture all three of scraper.overview, scraper.google.search, and scraper.aimode on the same query, same country, same timestamp.

Universal Scraping API for the rest of the LLM-answer landscape

Google AI Overviews are one share-of-citation surface. For a brand-level GEO program you usually also need to monitor ChatGPT search results, Perplexity answers, and other LLM-powered search experiences. The Universal Scraping API is the dedicated path for those — same x-api-token, different actor names, same JSON-envelope shape.

scraper.amazon (Rufus) for the commerce side

When the brand being monitored is a physical product, Amazon Rufus is the other end of the AI-answer pipeline. The Amazon Rufus actor returns the conversational shopping assistant's grounded answer for any natural-language product query. Combined with the Google AIO, it tells you how the two largest AI-answer surfaces position your product or your competitors'.

All four actors share one Scrapeless account, one x-api-token header, and one envelope shape. Wire a single client wrapper once and reuse it across the family.

How to avoid common problems

Error responses you might see

The API returns structured JSON for every error case. The code field, when present, is the Scrapeless error code; message is the human-readable explanation.

144xx codes are authentication and actor validation; the Field validation payload is request-shape; execution failed covers both no-AIO and transient upstream cases. Always store the task_id from any failed response — it is what Scrapeless support needs to correlate a single bad execution.

Problem-solution pairs

Problem: A query returns execution failed once but works on retry.
Solution: this is the transient upstream signal. The same query that fails now usually succeeds five to thirty seconds later. Retry with a 3–6 second back-off and a small attempt budget (3 tries). Treat persistent failure across that budget as "no AIO available for this query in this geography" rather than an integration bug.

Problem: A query consistently returns execution failed regardless of retry.
Solution: not every Google query produces an AI Overview. Single-word queries, navigational queries ("facebook"), and queries Google has not chosen to ground typically do not. Verify by opening https://www.google.com/search?q=<query> from the same geography (residential VPN or a real device); if no AIO renders for a human user, the actor will not produce one either. Re-phrase the query to be more informational ("how does X work", "best X for Y", "X vs Y").

Problem: task_result.products is null even though is_shopping is true.
Solution: task_result.products is populated for some shopping AIOs and null for others — even when is_shopping is true. When present, each item carries name, price, orig_price, discount, rating, review_count, seller, img, url, delivery, stores, and section_title (several string fields may be empty when Google did not surface that attribute). When the field is null, read product candidates from task_result.source (the cited sources panel — often retailer pages) and parse the markdown body in task_result.content for the inline product mentions. Always code defensively against null: result.get("products") or [].

Problem: task_result.source and task_result.web_source look similar — which is which?
Solution: source is the AI Overview's own citation panel (the links the AIO is attributed to). web_source is the related-search panel rendered below the AIO. For GEO citation tracking, count from source; for related-content opportunities, use web_source.

Problem: The AIO body in content has markdown citation refs ([1], [2]) but I need plain prose.
Solution: use task_result.rawtext — it is the citation-stripped twin, suitable for embeddings, downstream LLM prompts, and search-index ingestion. Use content when you need to render the answer with attribution.

Problem: The same query returns different AIO bodies on different calls.
Solution: AI Overviews are non-deterministic — Google regenerates them per session and they drift over hours and days. For monitoring use cases, pin the timestamp and country on every capture and keep the raw response; treat the body as a sample, not a constant. The cited-source set tends to be more stable than the prose body.

Problem: Rate-limit and concurrency caps are not in the public docs.
Solution: start with serial calls. Scale concurrency gradually while watching for HTTP 429 or a spike in execution failed. For sustained high-volume pipelines, contact Scrapeless support for a dedicated lane.

Conclusion: structured AIOs as a one-line dependency

Google's AI Overview has become a load-bearing surface for SEO, GEO, brand monitoring, and AI-grounding pipelines. Scraping it via DOM automation works but carries the full selector-maintenance, residential-egress, lazy-load, and CAPTCHA-handling stack underneath. The Scrapeless Scraper AI Overview API reduces the entire integration to one HTTP POST with three required fields (actor, input.prompt, input.country) and returns a structured JSON envelope where the AIO body, the cited sources, the related panel, the ads above the AIO, and the shopping flags are all first-class fields.

Paired with scraper.google.search for the organic SERP and scraper.aimode for Google's AI Mode tab, the three actors together cover Google's full AI-augmented search surface from a single Scrapeless account. The companion Universal Scraping API extends the same pattern to ChatGPT, Perplexity, and other LLM-answer surfaces — the building blocks of a production GEO program.

Sign up at app.scrapeless.com for free Scraper API credits, and read the full API reference at apidocs.scrapeless.com.

Ready to Build Your AI-Powered Search Pipeline?

Join our community to claim a free plan and connect with developers building GEO, brand-AI, and search-monitoring pipelines on top of Scrapeless:
Discord
Telegram

Sign up at Scrapeless Webiste for free Scraper API credits and adapt the patterns above to the keywords, countries, and brand terms your pipeline needs.

FAQ

Q1: Is scraping Google AI Overviews legal?
Public AI Overview content surfaced on google.com is part of the publicly visible search result and broadly considered fair to access for research, SEO monitoring, and competitive analysis. Specific jurisdictions and use cases differ — commercial use, redistribution of the AIO body, and at-scale automated access may carry additional considerations under Google's Terms of Service and local data-protection law. Review Google's ToS and your local regulations, and consult counsel before publishing or redistributing scraped AIO content.

Q2: Does every Google query produce an AI Overview?
No. Google chooses when to surface an AIO and the rate varies by query class, geography, language, and ongoing product experiments. Informational, comparison, "how does X work", "best X", and shopping-intent queries are the most likely to trigger one in 2026. Navigational queries ("facebook", "amazon login") and single-word ambiguous queries usually do not. The actor returns execution failed for queries Google does not ground in that geography.

Q3: Which countries are supported in input.country?
The country field accepts ISO 3166-1 alpha-2 codes and is verified end-to-end against US and GB. The residential-proxy footprint backing the actor spans 195+ countries, so most common codes (DE, FR, JP, CA, AU, BR, IN, ES, IT, NL) work; an unsupported code returns execution failed. Verify against your specific country with a small smoke test before scaling.

Q4: How fast is a single call?
End-to-end latency is typically 12–18 seconds in verification. Most of that is server-side polling for the AI Overview to settle behind Google's "generating" placeholder — the actor returns only the rendered body, not the placeholder.

Q5: Can I batch many queries concurrently?
Yes. Start with three to five parallel workers per token and scale up while watching the execution failed rate. Public rate-limit and concurrency caps are not documented; contact Scrapeless support for high-volume lanes.

Q6: How is scraper.overview different from scraper.google.search and scraper.aimode?
scraper.overview returns the AI Overview block specifically — body, citations, related panel, sponsored placements, shopping flags. scraper.google.search returns the rest of the SERP — organic ten blue links, Featured Snippet, People Also Ask, Knowledge Panel, Related Searches. scraper.aimode returns Google's separate AI Mode conversational page. For a complete view of how Google's AI Search presents a topic, capture all three on the same query, same country, same timestamp.

Q7: Can I get the structured product list for shopping AIOs?
Yes, for some shopping AIOs. When populated, task_result.products is an array of {name, price, orig_price, discount, rating, review_count, seller, img, url, delivery, stores, section_title} (several string fields may be empty when Google did not surface that attribute). It is null for other shopping AIOs even when is_shopping is true — in verification, 1 of 5 shopping captures returned a 10-item array; the rest were null. When the field is null, read product candidates from task_result.source (often retailer pages) and parse the inline mentions in task_result.content. Always code defensively against null (result.get("products") or []).

Q8: What does is_overview_shopping mean and how does it differ from is_shopping?
is_shopping is true when the underlying SERP carried a shopping module (the carousel above the organic results). is_overview_shopping is true when Google classified the AI Overview itself as shopping-oriented. purchase_link is true when the AIO body contains at least one direct purchase URL. Use the combination to filter for commercial-intent AIOs in retail-intel pipelines.

Q9: Do I need to handle the lazy-load placeholder myself?
No. AIOs render behind a "generating" placeholder, and the actor polls server-side until the body settles before returning. That polling is the dominant component of the ~12–18 s end-to-end latency. Callers do not implement a wait loop.

Q10: How should I store the response for long-term monitoring?
Full API reference and live request playground: apidocs.scrapeless.com. SDK and integration docs: docs.scrapeless.com.