Scraping Next.js Sites: Read the React Server Component Payload

TL;DR:

• A Next.js page ships its own data inside the HTML document. Parsing that payload gives you typed values directly, instead of scraping text back out of rendered markup.
• The App Router does not emit __NEXT_DATA__. It streams a React Server Component payload through repeated self.__next_f.push([1, "…"]) calls, and three live checks — nextjs.org, vercel.com, and a Scrapeless article page — all used that shape and none carried __NEXT_DATA__.
• The payload is a row stream, not one JSON object. Rows look like id:tag, where I rows reference client modules and T rows carry length-prefixed text.
• The T length is a hexadecimal count of UTF-8 bytes. Slicing by characters overran one real row by 36 bytes and pulled the next row's header into the extracted value.
• The payload can hold more than the page shows. On the article used here it carried the complete source Markdown, 11,025 characters with 8 headings and 6 code fences, plus the page's JSON-LD record.
• Start free. The rendering and fetching step used in the pivot runs on a free Universal Scraping API tier.

Server-rendered React has a side effect that is useful if you collect data: the server has to send the client everything it used to build the page. That data travels in the HTML document, next to the markup it produced.

Most guides for this still describe __NEXT_DATA__, a single JSON script tag. Sites on the App Router do not emit it. They stream something else, in a format that breaks a naive JSON parse — which is why a page can look like it has an obvious data blob and still resist every attempt to read it.

This guide reads that stream with the Python standard library, on a live page, and shows what comes out.

What Next.js Puts in the Document

Both Next.js routers embed their data, but the shape changed completely between them.

The App Router format is the wire representation of a rendered React Server Components tree. Components that run only on the server are serialized into this stream so the browser can reconstruct the page without re-fetching the data, a split the Next.js server and client component documentation describes in detail.

The practical consequence is that the data arrives in the initial response, so a single HTTP request is enough — without a headless browser, a wait for a client-side fetch, or an intercepted internal API.

Check Which Payload a Site Uses

Nothing here needs a third-party package — re, json, and urllib are enough. Check the shape first, because it decides everything downstream:

import re
import urllib.request

SITES = [
    "https://nextjs.org/",
    "https://vercel.com/",
    "https://www.scrapeless.com/en/blog/parsel-web-scraping",
]
BROWSER_UA = ("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
              "(KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36")


def detect(url):
    request = urllib.request.Request(url, headers={"User-Agent": BROWSER_UA})
    with urllib.request.urlopen(request, timeout=60) as response:
        html = response.read().decode("utf-8", "ignore")
    pages_router = '__NEXT_DATA__' in html
    app_router = 'self.__next_f' in html
    chunks = len(re.findall(r'self\.__next_f\.push\(', html))
    return pages_router, app_router, chunks


for url in SITES:
    pages_router, app_router, chunks = detect(url)
    shape = "app router (flight)" if app_router else ("pages router (__NEXT_DATA__)" if pages_router else "not next.js")
    print(f"{url}\n  __NEXT_DATA__={pages_router} self.__next_f={app_router} chunks={chunks} -> {shape}")

Running it against those three:

https://nextjs.org/
  __NEXT_DATA__=False self.__next_f=True chunks=48 -> app router (flight)
https://vercel.com/
  __NEXT_DATA__=False self.__next_f=True chunks=6 -> app router (flight)
https://www.scrapeless.com/en/blog/parsel-web-scraping
  __NEXT_DATA__=False self.__next_f=True chunks=75 -> app router (flight)

All three stream the flight payload, and the chunk counts differ by more than a factor of ten — 6 on one page, 75 on another. Chunk count tracks how much the server streamed, not how much data the page holds, so it is a signal that the payload exists rather than a measure of its size.

Reassemble the Stream

The payload is split across many inline <script> elements, each one calling self.__next_f.push with a fragment. Splitting content across script tags this way is ordinary DOM behavior under the HTML Standard's scripting rules; the browser executes them in order and the array accumulates.

To read it, capture every fragment and join them:

import json
import re

FLIGHT_CHUNK = re.compile(r'self\.__next_f\.push\(\[1,\s*"((?:[^"\\]|\\.)*)"\]\)')


def flight_payload(html):
    """Concatenate every streamed chunk back into one flight document."""
    chunks = FLIGHT_CHUNK.findall(html)
    return "".join(json.loads('"' + chunk + '"') for chunk in chunks), len(chunks)

The pattern keeps (?:[^"\\]|\\.)* rather than .*?, so an escaped quote inside the fragment does not end the match early. Each fragment is also a JSON string literal, so wrapping it in quotes and handing it to json.loads performs exactly the unescaping the browser would, which avoids hand-rolled \\n replacement.

On the article page, 75 chunks joined into a 303,530-character document.

Read the Rows

The reassembled payload is line-oriented. Each row starts with a hexadecimal id, a colon, and then a tag that says what follows:

1:"$Sreact.fragment"
2:I[1311,[],"default"]
65:T2b35,### TL;DR:

I rows are client module references. Rows beginning with { or [ are JSON. And T rows are text, prefixed with a length and a comma — those carry the interesting content. The third row above declares 2b35 bytes, which is 11,061 in decimal, and its text begins with the article's own opening heading.

TEXT_ROW = re.compile(rb'(?:^|\n)([0-9a-f]+):T([0-9a-f]+),')


def text_rows(payload):
    """Return {row_id: text}. The T prefix is a hex UTF-8 BYTE length, so slice bytes."""
    raw = payload.encode("utf-8")
    rows = {}
    for match in TEXT_ROW.finditer(raw):
        row_id = match.group(1).decode()
        length = int(match.group(2), 16)
        rows[row_id] = raw[match.end(): match.end() + length].decode("utf-8")
    return rows

The Length Prefix Counts Bytes, Not Characters

That function works on bytes for a reason, and it is the detail that quietly breaks a character-based implementation.

T declares its length as a hexadecimal number, and that number counts UTF-8 bytes. Any character outside ASCII — a curly quote, an em dash, an accented name — occupies more than one byte under the UTF-8 encoding specification, so a slice of characters of the same count reaches further into the stream than the row actually extends.

Measured on the article page, the Markdown row is 11,025 characters and 11,061 bytes. A character slice using the declared length ran 36 bytes past the end of the row and captured the beginning of the next row's header. Nothing raises an exception; the extracted value simply has junk on the end.

Encoding once and slicing bytes removes the class of bug entirely.

What the Payload Actually Contains

Nine text rows came back from the article page. Six held SVG path geometry, one held a marketing string, and two were worth having:

• the page's JSON-LD record — @type Article, headline parsel Web Scraping: CSS and XPath Selectors in Python, author Alex Johnson — the vocabulary defined by the schema.org Article type;
• the article's complete source Markdown: 11,025 characters, 8 ## headings, 6 fenced code blocks.

The Markdown is the point. The rendered DOM has that content as HTML, with the original heading levels, list structure, and code fences already compiled away. The flight payload carries the source, which is a materially better input for indexing, diffing, or feeding a model. For structured records embedded as ordinary script tags instead, the JSON-LD extraction guide covers the simpler case.

When the Document Itself Is Refused

This technique has one dependency: you need the initial HTML. That is where it runs out on its own.

A plain urllib.request.urlopen with the default Python-urllib/3.12 user agent returned HTTP 403 on the same article page that a browser user agent fetched at 444,510 characters. The payload was never the problem — the document never arrived.

Routing the fetch through the Universal Scraping API returns the document, and every step above then runs unchanged:

import json
import os
import urllib.request

UNLOCKER = "https://api.scrapeless.com/api/v2/unlocker/request"


def fetch_scrapeless(url):
    body = json.dumps({
        "actor": "unlocker.webunlocker",
        "input": {"url": url, "proxy_country": "US", "js_render": False},
    }).encode()
    request = urllib.request.Request(
        UNLOCKER,
        data=body,
        headers={
            "Content-Type": "application/json",
            "x-api-token": os.environ["SCRAPELESS_API_KEY"],
        },
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        envelope = json.loads(response.read().decode())
    return envelope["data"]

js_render is false here on purpose. The flight payload is in the server's first response, so rendering the page would cost time to produce data you already have. Turn it on only when content genuinely arrives after load — the page rendering guide covers that case, and parameters are in the Universal Scraping API reference.

No card is needed to try it — the free plan covers a run this size.

Put It Together

The complete extractor, fetch through parse:

import json
import os
import re
import urllib.error
import urllib.request

ARTICLE = "https://www.scrapeless.com/en/blog/parsel-web-scraping"
UNLOCKER = "https://api.scrapeless.com/api/v2/unlocker/request"

FLIGHT_CHUNK = re.compile(r'self\.__next_f\.push\(\[1,\s*"((?:[^"\\]|\\.)*)"\]\)')
TEXT_ROW = re.compile(rb'(?:^|\n)([0-9a-f]+):T([0-9a-f]+),')


def flight_payload(html):
    """Concatenate every streamed chunk back into one flight document."""
    chunks = FLIGHT_CHUNK.findall(html)
    return "".join(json.loads('"' + chunk + '"') for chunk in chunks), len(chunks)


def text_rows(payload):
    """Return {row_id: text}. The T prefix is a hex UTF-8 BYTE length, so slice bytes."""
    raw = payload.encode("utf-8")
    rows = {}
    for match in TEXT_ROW.finditer(raw):
        row_id = match.group(1).decode()
        length = int(match.group(2), 16)
        rows[row_id] = raw[match.end(): match.end() + length].decode("utf-8")
    return rows


def fetch_direct(url):
    try:
        response = urllib.request.urlopen(url, timeout=60)
        return response.status, response.read().decode("utf-8", "ignore")
    except urllib.error.HTTPError as exc:
        return exc.code, ""


def fetch_scrapeless(url):
    body = json.dumps({
        "actor": "unlocker.webunlocker",
        "input": {"url": url, "proxy_country": "US", "js_render": False},
    }).encode()
    request = urllib.request.Request(
        UNLOCKER,
        data=body,
        headers={
            "Content-Type": "application/json",
            "x-api-token": os.environ["SCRAPELESS_API_KEY"],
        },
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        envelope = json.loads(response.read().decode())
    return envelope["data"]


def main():
    status, _ = fetch_direct(ARTICLE)
    print(f"plain urllib, default user agent: HTTP {status}")

    html = fetch_scrapeless(ARTICLE)
    print(f"scrapeless universal scraping api: {len(html)} chars")

    payload, chunk_count = flight_payload(html)
    print(f"flight chunks: {chunk_count}")
    print(f"reassembled payload: {len(payload)} chars")

    rows = text_rows(payload)
    print(f"text rows: {len(rows)}")

    markdown = next(t for t in rows.values() if t.lstrip().startswith("###"))
    linked = json.loads(next(t for t in rows.values() if t.lstrip().startswith('{"@context"')))

    print(f"markdown row: {len(markdown)} chars, {len(markdown.encode())} bytes")
    print(f"markdown h2 headings: {len(re.findall(r'^## ', markdown, re.M))}")
    print(f"markdown code fences: {markdown.count('```') // 2}")
    print(f"json-ld type: {linked['@type']}")
    print(f"json-ld headline: {linked['headline']}")
    print(f"json-ld author: {linked['author']['name']}")

    # Slice the same row by characters to show why the byte length matters.
    raw = payload.encode("utf-8")
    match = next(m for m in TEXT_ROW.finditer(raw)
                 if raw[m.end():m.end() + 3].decode("utf-8", "ignore").startswith("###"))
    length = int(match.group(2), 16)
    start_char = len(raw[: match.end()].decode("utf-8"))
    by_char = payload[start_char: start_char + length]
    print(f"char slice overruns by: {len(by_char.encode()) - length} bytes")
    print(f"char slice leaks next row: {bool(re.search(chr(10) + '[0-9a-f]+:', by_char[-80:]))}")
    print(f"byte slice ends on article text: {markdown.rstrip().endswith('.')}")


if __name__ == "__main__":
    main()

Its output:

plain urllib, default user agent: HTTP 403
scrapeless universal scraping api: 444510 chars
flight chunks: 75
reassembled payload: 303530 chars
text rows: 9
markdown row: 11025 chars, 11061 bytes
markdown h2 headings: 8
markdown code fences: 6
json-ld type: Article
json-ld headline: parsel Web Scraping: CSS and XPath Selectors in Python
json-ld author: Alex Johnson
char slice overruns by: 36 bytes
char slice leaks next row: True
byte slice ends on article text: True

Troubleshooting

self.__next_f is absent but the site is clearly Next.js. The site is on the Pages Router. Read <script id="__NEXT_DATA__" type="application/json"> and parse its contents with json.loads; the row grammar in this post does not apply.

The regular expression matches nothing on a page you can see the payload in. The pattern requires the exact push([1, " prefix. Some chunks use other leading integers for different record kinds, and whitespace inside the call varies. Widen the pattern to self\.__next_f\.push\(\[\d+, and inspect what the other kinds contain before assuming they are text.

An extracted row ends in a stray 2:["$","div" fragment. The slice used characters where the format counts bytes. Encode the payload once and index the bytes object.

json.loads fails on a row that looks like valid JSON. Confirm you stripped the id: prefix and, for a T row, the T<length>, header as well. The row body starts after the comma, not after the colon.

Row ids move between deployments. They are stream positions, not stable identifiers. Select rows by content — a prefix test such as startswith('{"@context"') — rather than by hard-coding 60 or 65.

Conclusion

Read the document before reaching for a browser. A server-rendered React page has already done the data-fetching work and shipped the result inline, and parsing that is cheaper and more faithful than reconstructing values from rendered markup.

The format asks for a little care: reassemble the chunks, respect the row grammar, and count bytes where the format counts bytes. In exchange, a single HTTP response yielded a typed JSON-LD record and the article's original Markdown, with no rendering step anywhere in the pipeline.

Where a page genuinely builds its content after load, the data is not in the document and this technique does not apply — request interception is the tool for that shape.

Ready to run this against your own targets? Create a free Scrapeless account, export your key, and point the extractor at a page you already have permission to collect. Plan limits are on the pricing page.

FAQ

Q: How do I tell whether a site uses the App Router or the Pages Router?

Search the HTML source for two strings. self.__next_f means the App Router and the streamed flight payload; __NEXT_DATA__ means the Pages Router and a single JSON script tag. A site can serve both during a migration, with different routes on different shapes, so check the specific page you are collecting rather than the home page.

Q: Why can I see the JSON in the browser but not parse it in Python?

Because it is not one JSON document. The App Router payload is a sequence of rows spread across many script tags, and a single json.loads over the whole thing fails on the very first row. Concatenate the pushed fragments first, then parse row by row.

Q: What do the I rows mean?

I rows are client module references — an id, a chunk list, and an export name that tell the browser which JavaScript bundle to load for an interactive component. They contain no page data, so a data extractor skips them.

Q: Is the flight format stable enough to build on?

Treat the row grammar as reasonably stable and the row ids as disposable. The id:tag structure and the T<hex-length>, prefix have been consistent across the pages checked here, but ids are stream positions that change whenever the page's component tree changes. Select rows by their content, and assert on what you extracted before writing it anywhere.

Q: Does reading the payload avoid needing a browser entirely?

For data present in the first response, yes — that is the whole advantage, and it is why js_render stays off in the example. Content that a page fetches after load is not in the payload at all, and neither is anything behind an interaction, so those still need a rendered session.

Q: Is it legal to extract this?

The payload is part of the public HTML response, so the same rules apply as to any page you request: honor the site's terms and its robots directives, keep request volumes modest, collect only public information, and handle any personal data under the laws that apply to you. The page used throughout this guide is a public Scrapeless article.