Scraping Server-Sent Events (SSE) Streams With Scrapeless

A live blog's comment counter, a support widget's "agent is typing" indicator, and an AI chat reply that fills in word by word all share one trait: the browser opened the connection once, and the server has been pushing every update down that same open response ever since. There was no second request for the comment counter, no polling loop for the typing indicator — one HTTP GET, held open, with Content-Type: text/event-stream, and the server writing data: {...}\n\n onto it whenever something changes. A tool that fetches the page once and moves on never sees any of that, because the data arrives after the initial response headers, on a connection that never closes.

Connect Playwright to the Scrapeless Scraping Browser over wss://browser.scrapeless.com/api/v2/browser, and the CDP session underneath gives you two separate ways to read those pushed frames as they arrive: Playwright's own streamed response reader, and the raw Network.eventSourceMessageReceived event from the Chrome DevTools Protocol itself. This guide connects to that cloud browser, opens a real Server-Sent Events (SSE) stream, and captures frames both ways, with every code path run against a live public feed.

Why SSE Needs a Different Capture Path

page.goto() followed by a DOM read only shows whatever the page's markup contains at that moment. An SSE-fed widget never re-renders the whole page — it appends or replaces a fragment every time a new data: line lands, so a single DOM snapshot only catches whichever update happened to be current when you looked. The updates themselves never touch the DOM at all if nothing in the page bothers to render them; the only reliable place to read them is the stream itself.

SSE is also a narrower case than the two other real-time transports a browser can open. A hidden JSON endpoint answers one request with one response — read it with page.expect_response() and you are done. A WebSocket needs an Upgrade: websocket handshake before either side can send a frame, and once open it is full-duplex — either side can write at any time. SSE needs neither: the WHATWG Server-Sent Events specification defines it as a plain HTTP response whose body never ends, sent in response to an ordinary GET, readable with nothing more exotic than a streaming body reader. The server writes to it; the client only ever reads.

The Chrome DevTools Protocol exposes that stream directly, the same way it exposes WebSocket frames and intercepted HTTP responses. Its Network domain fires a dedicated eventSourceMessageReceived event for every message a page's EventSource connection receives, separate from the generic response-body events that fire for an ordinary fetch. The Scrapeless Scraping Browser is a cloud Chromium session reachable only over CDP — there is no WebDriver/Selenium endpoint to drive it with — so any CDP-capable client, Playwright here, can read either layer: the browser's own streamed body, or the protocol event underneath it.

Prerequisites

You need Python 3.9 or newer — playwright 1.59.0 declares Requires-Python >=3.9 on PyPI — the playwright package, and a Scrapeless API key from the free plan at app.scrapeless.com. No local Chrome binary is required: connect_over_cdp reaches a browser that already exists in Scrapeless's cloud.

The examples below connect to the Wikimedia Foundation's public recentchange stream, documented at Wikimedia's own EventStreams service page. It needs no API key and no account — every edit across every Wikimedia project is public by design, and the stream exists specifically so tools can consume it. Both examples stop themselves after five real frames, so neither run holds the connection open longer than it takes to prove the capture works.

Install

pip install playwright

export SCRAPELESS_API_KEY="your_scrapeless_api_key"

Connect Over CDP

Reuse the same URL-builder pattern any Playwright-to-Scraping-Browser script uses: three query parameters on one WSS endpoint.

import os
from urllib.parse import urlencode

API_KEY = os.environ["SCRAPELESS_API_KEY"]

def scraping_browser_url(proxy_country="US", session_ttl=60):
    params = urlencode({"token": API_KEY, "sessionTTL": session_ttl, "proxyCountry": proxy_country})
    return f"wss://browser.scrapeless.com/api/v2/browser?{params}"

Unlike a geofenced market-data socket, Wikimedia's public stream accepts connections from any region: both proxyCountry="US" and proxyCountry="DE" complete the handshake and start delivering frames in this session's live runs, with no close code or connection failure either way. proxyCountry still matters for plenty of real targets — a stream gated to a specific market is a real failure mode elsewhere in this series — it simply is not the constraint on this particular public feed. Confirm it against your own target rather than assuming either outcome.

Capture Frames With Playwright's Response Streaming

Playwright's response event API fires page.on("response") as soon as a response's headers arrive, without waiting for the body to finish — which matters here, because an SSE response never finishes on its own. Pair that with the page's own fetch() and a ReadableStream reader, and you can read the body as chunks land instead of waiting for a completion event that never comes:

import json
import os
from urllib.parse import urlencode

from playwright.sync_api import sync_playwright

API_KEY = os.environ["SCRAPELESS_API_KEY"]
FRAME_LIMIT = 5
STREAM_URL = "https://stream.wikimedia.org/v2/stream/recentchange"

def scraping_browser_url(proxy_country="US", session_ttl=60):
    params = urlencode({"token": API_KEY, "sessionTTL": session_ttl, "proxyCountry": proxy_country})
    return f"wss://browser.scrapeless.com/api/v2/browser?{params}"

responses_seen = []

def handle_response(response):
    if response.url == STREAM_URL:
        responses_seen.append((response.status, response.headers.get("content-type")))

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(scraping_browser_url())
    page = browser.new_page()
    page.on("response", handle_response)

    page.goto("about:blank")
    frames = page.evaluate(
        f"""async () => {{
            const resp = await fetch("{STREAM_URL}", {{ headers: {{ "Accept": "text/event-stream" }} }});
            const reader = resp.body.getReader();
            const decoder = new TextDecoder();
            let buffer = "";
            const out = [];
            while (out.length < {FRAME_LIMIT}) {{
                const {{ done, value }} = await reader.read();
                if (done) break;
                buffer += decoder.decode(value, {{ stream: true }});
                let idx;
                while ((idx = buffer.indexOf("\\n\\n")) !== -1 && out.length < {FRAME_LIMIT}) {{
                    const rawEvent = buffer.slice(0, idx);
                    buffer = buffer.slice(idx + 2);
                    const dataLine = rawEvent.split("\\n").find(l => l.startsWith("data:"));
                    if (dataLine) out.push(dataLine.slice(5).trim());
                }}
            }}
            await reader.cancel();
            return out;
        }}"""
    )
    browser.close()

print(f"response seen via page.on('response'): status={responses_seen[0][0]}, content-type={responses_seen[0][1]}")
print(f"captured {len(frames)} frames via in-page fetch() stream reader")
print(json.dumps(json.loads(frames[0]), indent=2))

Running it against the live feed prints a real Wikimedia edit event, along with the response Playwright itself observed:

response seen via page.on('response'): status=200, content-type=text/event-stream; charset=utf-8
captured 5 frames via in-page fetch() stream reader
{
  "$schema": "/mediawiki/recentchange/1.0.0",
  "meta": {
    "uri": "https://de.wikipedia.org/wiki/Liste_der_Kulturdenkmale_in_Oschatz",
    "domain": "de.wikipedia.org",
    "stream": "mediawiki.recentchange",
    "dt": "2026-07-28T14:16:13.757Z"
  },
  "id": 382817780,
  "type": "edit"
}

page.on("response") confirms Playwright's own network layer saw a 200 with the SSE content type on the outer HTTP response — proof this is one connection, not five separate requests. The in-page loop then reads that same connection's body one chunk at a time, splits on the blank line that the event-stream format uses to separate records, and pulls the data: line out of each one. reader.cancel() closes the underlying connection the moment five frames are in hand, so nothing here holds Wikimedia's stream open past what the proof needs.

Capture Raw Frames From the CDP Network Domain

Reading a fetch() stream by hand works, but it does not make Chrome's own network stack recognize the connection as an EventSource — that recognition is what fires the CDP eventSourceMessageReceived event, and it only fires when the page opens the stream with the browser's native EventSource object instead of a plain fetch() call. Reach for the raw CDP event when you want the browser's own accounting of the stream rather than a hand-rolled parser: it hands back eventName, eventId, and data as separate fields instead of raw text you have to split yourself.

import json
import os
from urllib.parse import urlencode

from playwright.sync_api import sync_playwright

API_KEY = os.environ["SCRAPELESS_API_KEY"]
FRAME_LIMIT = 5
STREAM_URL = "https://stream.wikimedia.org/v2/stream/recentchange"

def scraping_browser_url(proxy_country="US", session_ttl=60):
    params = urlencode({"token": API_KEY, "sessionTTL": session_ttl, "proxyCountry": proxy_country})
    return f"wss://browser.scrapeless.com/api/v2/browser?{params}"

cdp_events = []

def on_sse_message(event):
    cdp_events.append(event)

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(scraping_browser_url())
    page = browser.new_page()

    cdp = page.context.new_cdp_session(page)
    cdp.send("Network.enable")
    cdp.on("Network.eventSourceMessageReceived", on_sse_message)

    page.goto("about:blank")
    page.evaluate(
        f"""() => {{
            window.__count = 0;
            const es = new EventSource("{STREAM_URL}");
            window.__es = es;
            es.onmessage = () => {{
                window.__count += 1;
                if (window.__count >= {FRAME_LIMIT}) {{ es.close(); }}
            }};
        }}"""
    )
    page.wait_for_function(f"window.__count >= {FRAME_LIMIT}", timeout=20000)
    page.wait_for_timeout(300)
    browser.close()

print(f"captured {len(cdp_events)} raw CDP eventSourceMessageReceived events")
event = cdp_events[0]
print(f"eventName={event['eventName']}")
print(f"eventId={event['eventId']}")
print(json.dumps(json.loads(event["data"]), indent=2))

The raw protocol event carries the same edit payload, plus fields the fetch-based parser had to skip:

captured 5 raw CDP eventSourceMessageReceived events
eventName=message
eventId=[{"topic":"eqiad.mediawiki.recentchange","partition":0,"timestamp":1785248186774},{"topic":"codfw.mediawiki.recentchange","partition":0,"offset":-1}]
{
  "$schema": "/mediawiki/recentchange/1.0.0",
  "meta": {
    "uri": "https://www.wikidata.org/wiki/Q100886493",
    "domain": "www.wikidata.org",
    "stream": "mediawiki.recentchange",
    "dt": "2026-07-28T14:16:26.773Z"
  },
  "id": 2602974671,
  "type": "edit"
}

page.context.new_cdp_session(page) opens a session whose CDPSession interface exposes send() for protocol commands and on() for protocol events; Network.enable turns event reporting on, and every subsequent eventSourceMessageReceived event fires with the exact eventName/eventId/data shape the DevTools Network panel reads for an EventSource row. eventId here is not a plain counter — Wikimedia encodes Kafka topic, partition, and offset metadata into it, because that value doubles as a resume cursor: reconnect with it as the Last-Event-ID request header, and the service picks up from that exact position instead of replaying from the start.

What You Get Back

Both paths return the same underlying edit event for this stream, because both are reading the same open connection's pushed records — one through a hand-rolled parser over a generic fetch, one straight from the protocol's own SSE accounting.

Wikimedia's own documentation recommends filtering out records where meta.domain equals "canary" — synthetic heartbeat events the service injects for its own monitoring, not real edits. Any consumer of this feed should drop those before treating a record as user activity, the same way you would drop a keep-alive comment line (: keepalive\n\n) that some SSE servers send to hold idle connections open; the format permits an SSE line starting with : to be a comment with no data: field at all, and both capture paths above already skip it naturally since neither one looks for content there.

Get your API key on the free plan: app.scrapeless.com

SSE vs WebSocket in Practice

The two protocols solve overlapping problems with different tradeoffs, and picking the wrong one to look for costs real debugging time. A WebSocket needs a 101 Switching Protocols handshake before any frame moves and stays open full-duplex; per the WebSocket capture guide in this series, nothing reconnects a dropped WebSocket automatically. SSE answers a plain GET with a 200 and Content-Type: text/event-stream, only the server ever writes, and the browser's native EventSource object reconnects on its own. Per the WHATWG spec, if the connection closes the user agent waits an implementation-defined delay (commonly a few seconds, adjustable per-stream through a dedicated reconnection-delay field the format defines), then reopens the request, attaching Last-Event-ID automatically so the server can resume instead of replaying everything.

That reconnect behavior is why the CDP-level distinction in this guide matters in practice: a raw fetch() reader has to reimplement reconnection and Last-Event-ID tracking by hand, while a real EventSource object gets it for free from the browser — at the cost of losing direct control over exactly when a new connection opens. Read a target's traffic with a real EventSource when you want the browser to manage reconnection; read it with fetch() plus a stream reader when you need to control the connection lifecycle yourself, such as canceling after a bounded number of records the way both examples above do.

Reading an SSE Connection a Real Page Already Opened

Both capture methods above open the connection from an otherwise blank page, because that keeps the target small and public. A live dashboard, a chat UI, or a notification feed opens its own EventSource or streamed fetch() the same way, from its own bundled JavaScript, as soon as the relevant component mounts — and page.on("response") plus the CDP Network domain fire identically either way. Attach the same handlers before calling page.goto() on the real target instead of on about:blank, and frames arrive as the page's own script receives them; nothing about the capture logic changes because the connection happens to belong to the page instead of to a page.evaluate() call. What does change is discovery — open the target's own Network panel once, filter by EventSource or Fetch/XHR, and confirm the endpoint and its Content-Type before writing a handler around it, rather than guessing at a stream URL in advance.

Conclusion

An SSE connection is the plainest of the three real-time transports a browser can open — one GET, one open response, no handshake — and that plainness is exactly why a tool that only reads the DOM or waits for a response to finish never sees it. Playwright's page.on("response") with an in-page stream reader and the raw CDP Network.eventSourceMessageReceived event both read that same pushed data, one through a hand-rolled parser and one straight from the protocol, and both worked identically against a real public Wikimedia edit stream over the Scrapeless Scraping Browser's CDP connection. Cap how many records you capture before closing the connection, respect the resume cursor a stream's id: field carries if you reconnect, and the rest of the script is the handful of Playwright calls this guide already walked through. Read the current session and egress limits on the Scraping Browser product page and check plan limits on the pricing page. For the connection mechanics this guide builds on, the Chrome DevTools Protocol explainer walks through what CDP exposes beyond the Network domain.

Join our community to claim a free plan and compare notes with other developers building browser automation: Discord · Telegram.

FAQ

Q: Do I need Selenium or WebDriver to capture SSE frames this way?

No. The Scrapeless Scraping Browser is reachable only over the Chrome DevTools Protocol, so any client that speaks CDP — Playwright here, or Puppeteer — can connect and read the stream. There is no WebDriver endpoint, so Selenium cannot drive this connection.

Q: Why doesn't a plain fetch() reader trigger the CDP eventSourceMessageReceived event?

Chrome's network stack only classifies a connection as an EventSource, and reports it through that dedicated event, when the page opens it with the native EventSource object. A fetch() call streams bytes the same way at the transport level, but Chrome does not parse or tag it as SSE, so reading it requires parsing the text/event-stream format yourself, as the response-streaming example in this guide does.

Q: Does the SSE connection reconnect automatically if it drops?

Only when opened with the native EventSource object. Per the WHATWG specification, the browser waits an implementation-defined delay and then reopens the request with a Last-Event-ID header set to the last id: value it saw, so a well-behaved server can resume instead of replaying from the start. A fetch()-based reader gets none of this automatically — reconnection and Last-Event-ID tracking have to be written by hand.

Q: How is this different from the network-request-interception technique in this series?

Interception reads discrete request/response pairs — a page fires a new HTTP request every time it needs fresh data, and you catch each one. An SSE connection is a single request whose response body never finishes; there is nothing to intercept repeatedly, because the server keeps writing to the one response it already sent.

Q: What happens with : comment lines or the canary events on Wikimedia's stream?

A line starting with : in the SSE format is a comment with no data: field, sent by some servers to keep an idle connection alive; both capture paths in this guide only look for data: lines, so a comment line is skipped automatically. Wikimedia also injects synthetic meta.domain: "canary" records for its own monitoring — filter those out before treating a record as a real edit.

Q: Is it safe to run this against any SSE endpoint I find?

Only against public, unauthenticated endpoints you are permitted to read, and only at a volume the endpoint's own documentation allows. The example here targets Wikimedia's documented public edit stream, needs no key, and closes the connection after five records rather than holding it open indefinitely.

Q: How long can the SSE connection stay open?

The sessionTTL query parameter bounds the browser session in seconds. A short value is enough for a bounded capture like the one in this guide; a longer one keeps the session — and any open stream connections — alive for a longer-running consumer.

Q: Can I read an SSE stream without a browser at all?

Yes, for a public unauthenticated endpoint like this one — a plain HTTP client that supports chunked reads can parse the same text/event-stream format directly. The browser session in this guide is worth it when the target requires a real Chromium fingerprint to establish the connection in the first place, or when a page opens the stream itself as a side effect of its own JavaScript rather than exposing a documented standalone endpoint.