Playwright + Scrapeless Scraping Browser: Capture and Replay a Hidden GraphQL API

Open the Network tab on rickandmortyapi.com/graphql and watch it for a minute: the schema-introspection call GraphiQL fires the instant the page loads, and the query you type in and run yourself a moment later, both land on the exact same URL. A REST API spreads its behavior across paths — /characters, /episodes, /locations/1 — so the URL alone tells you what a request is for. A GraphQL API collapses all of that into one endpoint and moves the actual request into the POST body instead: a query string naming the fields you want, a variables object supplying the arguments, sometimes an operationName tag identifying which one this is. Reading that traffic means reading the body, not the URL, because the URL stopped carrying the signal.

This guide connects Playwright to the Scrapeless Scraping Browser over CDP, drives a real public GraphQL playground into firing a query, and intercepts the resulting POST two independent ways — Playwright's own response events, and the raw CDP Network domain underneath them — before replaying that exact request with a plain HTTP client and no browser at all. Every command below ran against the live target.

One Endpoint, Every Operation

https://rickandmortyapi.graphcdn.app/ is the address the Rick and Morty API's own GraphiQL playground actually calls, one layer behind the friendlier rickandmortyapi.com/graphql alias its documentation advertises; it answers both requests to that alias and requests to the CDN address directly with identical data. That single address serves every operation the playground can send: the introspection query it fires automatically on load to populate its schema explorer, and whatever query you type in and execute yourself. A network filter written against that URL alone (page.route("**/graphcdn.app/**", ...), or a CDP listener keyed only on hostname) would catch both indiscriminately — exactly the problem GraphQL's own HTTP-serving convention creates by design: one URL, one method, every operation distinguished by what's inside the request instead of where it's sent. Isolating the one query that actually matters means reading the POST body's operationName field or the query text itself, not the address it went to.

The playground itself makes the second half of the difference obvious: unlike a scroll-triggered REST endpoint that fires the moment a page loads or a user scrolls, GraphiQL's query editor starts empty. Nothing meaningful happens until you type a query and click Execute — the technique here has to drive that interaction, not just wait for it.

Prerequisites

You need Python 3.9 or newer — playwright 1.59.0 declares Requires-Python >=3.9 on PyPI — the playwright package, and a Scrapeless API key from the free plan at app.scrapeless.com. The target GraphQL endpoint itself needs no key or account of its own; it is public, unauthenticated data. Keep the Scrapeless key in an environment variable rather than a literal in your script, since it travels as the token query parameter on the Scraping Browser's CDP endpoint.

Install

pip install playwright

export SCRAPELESS_API_KEY="your_scrapeless_api_key"

Connect Over CDP

Reuse the same URL-builder pattern every Playwright-to-Scraping-Browser script in this series uses: three query parameters on one WSS endpoint.

import os
from urllib.parse import urlencode

API_KEY = os.environ["SCRAPELESS_API_KEY"]

def scraping_browser_url(proxy_country="US", session_ttl=120):
    params = urlencode({
        "token": API_KEY,
        "sessionTTL": session_ttl,
        "proxyCountry": proxy_country,
    })
    return f"wss://browser.scrapeless.com/api/v2/browser?{params}"

chromium.connect_over_cdp(scraping_browser_url()) hands back a standard Playwright Browser object, no local Chrome install required. Nothing about the two interception techniques below is Scraping-Browser-specific — they run against any CDP-reachable Chromium — but running the render on Scraping Browser's infrastructure means a GraphQL frontend that fingerprints its own client still hydrates and fires its queries normally.

Trigger the Query and Capture It With a Response Listener

page.expect_response() binds the wait to the action that triggers it, so it works whether that action is a page.goto() or, as here, a UI interaction you drive yourself. Type a real query and its variables into GraphiQL's editor, click Execute inside the expect_response context, and the captured Response object hands back exactly what the site's own JavaScript sent and received:

import json
import os
from urllib.parse import urlencode

from playwright.sync_api import sync_playwright

API_KEY = os.environ["SCRAPELESS_API_KEY"]

QUERY = (
    "query GetCharacters($page: Int, $name: String) { "
    "characters(page: $page, filter: { name: $name }) { "
    "info { count pages } "
    "results { id name status species } } }"
)
VARIABLES = '{"page": 1, "name": "rick"}'

def scraping_browser_url(proxy_country="US", session_ttl=120):
    params = urlencode({
        "token": API_KEY,
        "sessionTTL": session_ttl,
        "proxyCountry": proxy_country,
    })
    return f"wss://browser.scrapeless.com/api/v2/browser?{params}"

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(scraping_browser_url())
    page = browser.new_page()
    page.goto("https://rickandmortyapi.com/graphql", wait_until="domcontentloaded")

    query_editor = page.locator(".graphiql-query-editor .CodeMirror").first
    query_editor.click()
    page.keyboard.press("Control+A")
    page.keyboard.insert_text(QUERY)

    page.locator("button:has-text('Variables')").first.click()
    variables_editor = page.locator(".graphiql-editor-tool .CodeMirror").first
    variables_editor.click()
    page.keyboard.press("Control+A")
    page.keyboard.insert_text(VARIABLES)

    with page.expect_response(lambda r: "graphcdn.app" in r.url and r.request.method == "POST") as run:
        page.locator("button.graphiql-execute-button").click()

    resp = run.value
    sent = json.loads(resp.request.post_data)
    data = resp.json()["data"]["characters"]

    print("POST target:", resp.request.url)
    print("operationName:", sent["operationName"])
    print("variables sent:", sent["variables"])
    print("info:", data["info"])
    print("first result:", data["results"][0])
    print("result count in this page:", len(data["results"]))

    browser.close()

Running it against the live playground prints:

POST target: https://rickandmortyapi.graphcdn.app/
operationName: GetCharacters
variables sent: {'page': 1, 'name': 'rick'}
info: {'count': 107, 'pages': 6}
first result: {'id': '1', 'name': 'Rick Sanchez', 'status': 'Alive', 'species': 'Human'}
result count in this page: 20

sent["variables"] is the same Python dict the editor's Variables panel held — {"page": 1, "name": "rick"} — confirming the interception read the actual request body, not a guess at what the query might contain. page.keyboard.insert_text() rather than page.keyboard.type() matters here: CodeMirror, the editor GraphiQL uses, auto-closes brackets as you type them character by character, so simulating individual keystrokes for a query full of { and } produces duplicated closing braces and a syntax error. insert_text() inserts the whole string at once, the way a paste would, and skips the per-keystroke auto-closing logic entirely.

Match the Right Request in the Raw CDP Network Domain

Playwright's response events sit on top of the Chrome DevTools Protocol Network domain, reachable directly through a CDPSession for cases where you're not driving Playwright at all — a bare CDP client, or a tool that only exposes protocol events. Because the endpoint URL alone doesn't distinguish operations, the CDP-level filter has to inspect postData the same way the higher-level capture implicitly does by matching the triggering click:

import json
import os
from urllib.parse import urlencode

from playwright.sync_api import sync_playwright

API_KEY = os.environ["SCRAPELESS_API_KEY"]

QUERY = (
    "query GetCharacters($page: Int, $name: String) { "
    "characters(page: $page, filter: { name: $name }) { "
    "info { count pages } "
    "results { id name status species } } }"
)
VARIABLES = '{"page": 2, "name": "rick"}'

captured = {}

def scraping_browser_url(proxy_country="US", session_ttl=120):
    params = urlencode({
        "token": API_KEY,
        "sessionTTL": session_ttl,
        "proxyCountry": proxy_country,
    })
    return f"wss://browser.scrapeless.com/api/v2/browser?{params}"

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(scraping_browser_url())
    page = browser.new_page()
    cdp = page.context.new_cdp_session(page)
    cdp.send("Network.enable")

    def on_request(event):
        # The playground also fires a schema-introspection POST to this same
        # URL on load. Matching on operationName in the body -- not the URL
        # -- is what separates it from the query this script triggers.
        request = event["request"]
        if "graphcdn.app" in request["url"] and "GetCharacters" in request.get("postData", ""):
            captured[event["requestId"]] = None

    def on_finished(event):
        request_id = event["requestId"]
        if request_id in captured and captured[request_id] is None:
            body = cdp.send("Network.getResponseBody", {"requestId": request_id})
            captured[request_id] = json.loads(body["body"])

    cdp.on("Network.requestWillBeSent", on_request)
    cdp.on("Network.loadingFinished", on_finished)

    page.goto("https://rickandmortyapi.com/graphql", wait_until="domcontentloaded")

    query_editor = page.locator(".graphiql-query-editor .CodeMirror").first
    query_editor.click()
    page.keyboard.press("Control+A")
    page.keyboard.insert_text(QUERY)

    page.locator("button:has-text('Variables')").first.click()
    variables_editor = page.locator(".graphiql-editor-tool .CodeMirror").first
    variables_editor.click()
    page.keyboard.press("Control+A")
    page.keyboard.insert_text(VARIABLES)

    page.locator("button.graphiql-execute-button").click()
    for _ in range(30):
        if captured and all(v is not None for v in captured.values()):
            break
        page.wait_for_timeout(300)

    data = next(iter(captured.values()))["data"]["characters"]
    print("requests matched by body content:", len(captured))
    print("info:", data["info"])
    print("first result:", data["results"][0])

    browser.close()

requests matched by body content: 1
info: {'count': 107, 'pages': 6}
first result: {'id': '218', 'name': 'Mechanical Rick', 'status': 'unknown', 'species': 'Robot'}

Network.requestWillBeSent fires with the outgoing request's own postData field already attached, before the response exists — the natural point to decide whether this particular POST is the one worth tracking. Network.loadingFinished confirms the matching response finished transferring, and only then does getResponseBody return the bytes. Page 2 comes back with a different first result than page 1 did, which is the point: the raw-CDP path and the response-listener path are reading the same wire, matched two different ways, and both land on real, distinct data from the same live query.

What You Get Back

Both capture paths return the same Character shape for this query, because both read the same underlying response.

Swap filter: { name: $name } for filter: { status: "Alive" } or drop the filter argument entirely, and the same two capture scripts keep working unmodified — only the variables payload and the resulting info.count change, because the wire-level technique doesn't depend on which fields or arguments a particular query happens to use.

Get free Scraping Browser runtime by signing up at app.scrapeless.com and running both capture scripts above against a GraphQL endpoint of your own.

Replay the Query With No Browser at All

Both interceptions above proved the same thing: https://rickandmortyapi.graphcdn.app/ accepts a plain JSON POST with query, variables, and operationName, no authentication, and returns the same Character data either capture already showed. Once that shape is known, a browser is no longer required to ask it the identical question — though the request has to declare a normal User-Agent, or the gateway's edge rejects it outright regardless of the payload; the limits section below covers why:

import json
import urllib.error
import urllib.request

ENDPOINT = "https://rickandmortyapi.graphcdn.app/"

QUERY = (
    "query GetCharacters($page: Int, $name: String) { "
    "characters(page: $page, filter: { name: $name }) { "
    "info { count pages } "
    "results { id name status species } } }"
)

payload = json.dumps({
    "query": QUERY,
    "variables": {"page": 1, "name": "rick"},
    "operationName": "GetCharacters",
}).encode("utf-8")

req = urllib.request.Request(
    ENDPOINT,
    data=payload,
    # A default urllib request declares "Python-urllib/x.y" as its User-Agent
    # and the gateway's edge rejects that outright -- see "When the Browser
    # Stays in the Loop" below for what's actually being checked.
    headers={
        "Content-Type": "application/json",
        "User-Agent": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
            "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
        ),
    },
    method="POST",
)

with urllib.request.urlopen(req, timeout=10) as resp:
    if resp.status != 200:
        raise urllib.error.HTTPError(ENDPOINT, resp.status, "unexpected status", resp.headers, None)
    body = json.loads(resp.read())

data = body["data"]["characters"]
print("status: 200, no browser process involved")
print("info:", data["info"])
print("first result:", data["results"][0])
print("result count in this page:", len(data["results"]))

status: 200, no browser process involved
info: {'count': 107, 'pages': 6}
first result: {'id': '1', 'name': 'Rick Sanchez', 'status': 'Alive', 'species': 'Human'}
result count in this page: 20

Same info, same first result, same 20-row page as the response-listener capture — because it's the exact same request, sent by urllib instead of by GraphiQL's own fetch call. The browser's entire job in this workflow was to reveal the endpoint, the query shape, and the variables format; once those are known, a GraphQL POST is just JSON over HTTP carrying a request document in the shape the GraphQL specification itself defines, and the fastest way to ask the same question again is usually to stop rendering a page and ask it directly.

When the Browser Stays in the Loop

Not every GraphQL endpoint is this cooperative, for reasons specific to how GraphQL gateways are commonly deployed. Plenty require an Authorization header carrying a token the frontend's own JavaScript attaches from local storage or a cookie, something you can't reconstruct unless you captured it from a real session — the same limitation REST-shaped hidden APIs share. Some go further and enforce Automatic Persisted Queries, where the client sends a SHA-256 hash of the query instead of the query text itself; a server that only accepts pre-registered hashes rejects a replayed request built from a query string alone, because the hash was never registered from that client. In both cases the interception step still works exactly as shown here: page.expect_response() and the CDP Network domain read whatever the browser actually sent, auth header or persisted-query hash included. Only the direct-replay payoff stops applying, because reconstructing what the browser attached becomes the hard part.

A subtler limit showed up during verification of this article and is worth naming directly: a public, unauthenticated GraphQL endpoint can still sit behind fingerprint-based bot mitigation that has nothing to do with the query itself. A plain urllib POST against the endpoint above, sent with no User-Agent header (Python's own default, literally the string Python-urllib/3.12), returned an HTTP 403 with Cloudflare error 1010: "the owner of this website has banned your access based on your browser's signature." That happened every time, reproducibly, even though the gateway's own query-cost rate-limit headers reported budget still remaining. Adding a single ordinary browser User-Agent string, and nothing else about the request, passed the same check on every following call. The block was keyed on the client's declared identity, not the request's content or how often it arrived. A cloud browser session that presents a real Chromium signature, the kind the Scraping Browser's CDP endpoint provides, never carries that mismatch in the first place.

Conclusion

A GraphQL API trades REST's many self-describing URLs for one endpoint and a request body that has to be read to know what it's asking. page.expect_response() and the raw CDP Network domain both read that body regardless, matched by content instead of address, and a plain HTTP client replays the same JSON once the shape is confirmed. Keep the query filter keyed on operationName or the query text rather than the URL, expect an empty query editor to need real typed input before anything interesting fires, and treat a public endpoint's bot-mitigation layer as a separate concern from its authentication. For the CDP mechanics both capture paths build on, the Chrome DevTools Protocol explainer walks through what the protocol exposes beyond the Network domain.

Sign up at app.scrapeless.com for free Scraping Browser runtime, or see the Scraping Browser product page and pricing for scaled runs.

Join our community to compare notes with other developers building browser automation: Discord · Telegram.

FAQ

Q: What is GraphQL interception in web scraping?
It's reading the single POST request a GraphQL-backed page's own JavaScript sends to fetch its data — the query and variables in that request body — instead of waiting for the response to render into HTML and parsing the markup back out.

Q: Why can't you tell which GraphQL operation ran just from the request URL?
Because a GraphQL gateway typically serves every operation from one fixed endpoint. Unlike a REST API, where different paths correspond to different resources, a GraphQL request's identity lives in its POST body — the operationName field or the query text — not in the address it was sent to.

Q: Do you need a browser once you know the query, variables, and endpoint?
Only if the endpoint requires something the browser supplies, such as an authorization header or a registered persisted-query hash. A public endpoint that accepts a full query string with no authentication, like the one in this guide, can be replayed with a plain HTTP client, as the direct-replay example shows.

Q: What's the difference between page.expect_response() and the raw CDP Network domain here?
page.expect_response() is Playwright's higher-level wrapper, bound to the action that triggers the request and returning a parsed Response object. The CDP Network domain is the protocol underneath it — Network.requestWillBeSent, Network.loadingFinished, and Network.getResponseBody — useful without a Playwright binding at all, or when a filter needs to inspect the outgoing request body before the response exists.

Q: Is intercepting a public GraphQL playground's own queries legal?
Reading responses your own browser session already receives while visiting a public page carries different considerations than reaching authenticated or non-public data. Scope any workflow to public pages, respect the target's terms of service and robots directives, and keep request volume bounded — interception is a way to read traffic precisely, not license to ignore access rules.

Q: What happens with an authenticated or persisted-query-only GraphQL API?
The interception step still works — both capture paths read whatever the browser actually sent, authorization header or persisted-query hash included. The direct-replay step is what breaks, because a hash-only server rejects a request built from a raw query string that was never registered, and an authenticated endpoint rejects a request missing the header the original session carried.

Q: Why does the direct-replay example set a User-Agent header if it isn't a browser?
Because the gateway's edge rejects the request without one. A plain urllib POST using Python's default User-Agent string returns a Cloudflare error 1010 on every attempt, even though the query-cost rate-limit headers show budget remaining — the block is keyed on the client's declared identity, not the query or how often it's sent. One ordinary browser User-Agent string, with nothing else about the request changed, is enough to pass.

Q: Does this technique need the Scrapeless Scraping Browser specifically, or does it work with any CDP-reachable Chromium?
The interception mechanics are generic CDP behavior and work against any Chromium reachable through connect_over_cdp, local or remote. Running them on the Scrapeless Scraping Browser adds a cloud Chromium session with a real browser signature, which matters for a frontend that fingerprints its client before letting a query fire in the first place.

Q: What happens if the target changes its schema or query shape?
The interception code keeps working as long as the endpoint URL still matches — it reads whatever body the browser sends regardless of the query's fields. A renamed field or restructured type breaks the code reading data["characters"]["results"], the same way a CSS selector breaks when a class name changes; a GraphQL schema is typically more stable than markup, but it isn't immune to a breaking change.