Vision LLM Web Scraping: Read the Screenshot, Not the DOM

A vision model does not read your HTML; it reads a picture of the page. That changes what "scraping" means: instead of hunting for the selector that holds a value, you hand a screenshot to a model and ask what it sees. The Scrapeless Scraping Browser takes that screenshot in the cloud over the Chrome DevTools Protocol, and a multimodal model turns the pixels into structured data.

This is the tool to reach for when the DOM fights you — a price baked into an image, a chart with no underlying table, a canvas-rendered widget, or a layout where the meaning lives in position rather than in tags. This guide connects to the Scraping Browser, captures a screenshot of a rendered page, and has a vision model return JSON from the image, with every step executed live.

Why read a screenshot instead of the DOM

Selector-based scraping assumes the data you want is a value in the markup. Often it is not. A storefront may render prices as images to deter bots, a dashboard may draw numbers onto a canvas element that carries no text, and a marketing page may encode a comparison entirely in visual layout. In those cases the HTML is either empty of the value or so tangled that the selector is more fragile than the thing it points at.

A vision model sidesteps the markup. It sees what a person sees — the rendered, laid-out page — and reads the values off the image. The screenshot has to be faithful, which is why it comes from a real browser: the Scraping Browser renders the page server-side, runs its scripts, and captures the frame a user would see, so the model reasons about the finished page rather than a half-built one.

Prerequisites

You need Python 3.9 or newer, the playwright client and requests, a Scrapeless API key for the browser session, and a key for a multimodal model. Get the Scrapeless key on the free plan at app.scrapeless.com. Keep both keys in environment variables; they travel in a connection URL and an authorization header, never in source.

Install

pip install playwright requests

Configure

export SCRAPELESS_API_KEY="your_scrapeless_api_key"
export OPENROUTER_API_KEY="your_model_api_key"

Capture the screenshot with the Scraping Browser

Connect to the cloud browser over CDP, open the page, and capture the viewport. The screenshot is a PNG produced by the same Chromium that rendered the page, so what the model receives is exactly what a visitor would see. The capture happens through the Chrome DevTools Protocol and returns bytes in the PNG image format:

import os
from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

def browser_url():
    return "wss://browser.scrapeless.com/api/v2/browser?" + urlencode(
        {"token": os.environ["SCRAPELESS_API_KEY"], "sessionTTL": 180, "proxyCountry": "US"})

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(browser_url())
    page = browser.new_page(viewport={"width": 1280, "height": 900})
    page.goto("https://quotes.toscrape.com/", wait_until="networkidle")
    png = page.screenshot()
    browser.close()

print("screenshot bytes:", len(png))
print("PNG signature ok:", png[:8] == b"\x89PNG\r\n\x1a\n")

The run confirms a real image came back:

screenshot bytes: 55462
PNG signature ok: True

Get your API key on the free plan: app.scrapeless.com

Read the screenshot with a vision model

Encode the PNG as a base64 data URL, send it to a multimodal model alongside the schema you want, and constrain the reply to JSON. The model reads the rendered page and returns the fields — no selectors, no DOM traversal. Base64 follows the Base64 data-encoding standard, which is how the image rides inside a JSON request. The full capture-and-read pipeline:

import os, base64, json, requests
from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

def browser_url():
    return "wss://browser.scrapeless.com/api/v2/browser?" + urlencode(
        {"token": os.environ["SCRAPELESS_API_KEY"], "sessionTTL": 180, "proxyCountry": "US"})

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(browser_url())
    page = browser.new_page(viewport={"width": 1280, "height": 900})
    page.goto("https://quotes.toscrape.com/", wait_until="networkidle")
    png = page.screenshot()
    browser.close()

img = "data:image/png;base64," + base64.b64encode(png).decode()
resp = requests.post(
    "https://openrouter.ai/api/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}", "Content-Type": "application/json"},
    json={"model": "google/gemini-2.5-flash-lite",
          "messages": [{"role": "user", "content": [
              {"type": "text", "text": 'From this screenshot of a quotes page, return JSON '
                                       '{"quotes":[{"author":..., "text":...}]} for the first 3 quotes '
                                       'you can see. Return ONLY JSON.'},
              {"type": "image_url", "image_url": {"url": img}}]}],
          "response_format": {"type": "json_object"}, "temperature": 0},
    timeout=120,
)
resp.raise_for_status()
quotes = json.loads(resp.json()["choices"][0]["message"]["content"])["quotes"]
print("quotes extracted from image:", len(quotes))
print("first author from image:", quotes[0]["author"])

The model reads the pixels and returns the records:

quotes extracted from image: 3
first author from image: Albert Einstein

The image was never parsed as HTML. The author name came from the rendered text in the screenshot, which is why this approach survives a page that hides the same value behind an image tag or a canvas.

When vision earns its cost, and when it does not

Vision scraping is not free — an image is far more tokens than the snippet of HTML that holds the same value, and the model can misread small or low-contrast text. So reach for it deliberately. It pays off when the value is genuinely visual: text baked into images, charts and canvas widgets, PDFs rendered to a page, or a layout whose meaning is spatial. When the data is clean text in the DOM, a selector is cheaper, faster, and exact, and reading it from the markup is the better choice. The strength of pairing the Scraping Browser with a vision model is that you can switch between the two against the same rendered session — parse the DOM where it is reliable, screenshot and read where it is not.

Conclusion

Vision-model scraping reads the page the way a person does. The Scrapeless Scraping Browser renders the target and captures a faithful screenshot in the cloud over CDP, and a multimodal model turns that image into structured JSON. The run above pulled three quotes and the author "Albert Einstein" from pixels, with no selector anywhere in the code. Use it where the DOM cannot carry the value, and keep selectors for the parts of a page that already hold clean text — the explainer on what an LLM scraper is frames where models fit in extraction. Read the capabilities on the Scraping Browser product page and weigh image volume against the pricing page before you run it at scale.

Join our community to claim a free plan and compare notes with other developers building extraction pipelines: Discord · Telegram.

FAQ

Q: When should I scrape by screenshot instead of parsing HTML?

Use vision when the value is visual rather than textual — a price rendered as an image, numbers drawn on a canvas, a chart with no underlying table, or a layout that encodes meaning by position. When the data is clean text in the DOM, a CSS selector is cheaper and more exact.

Q: Why capture the screenshot with the Scraping Browser rather than a local one?

Because the model can only read what the screenshot shows. The Scraping Browser renders the page server-side, runs its scripts, and captures the finished frame from cloud Chromium, so the image is the fully rendered page rather than a half-loaded one from a local headless browser you also have to maintain.

Q: How does the image get into the request?

The PNG bytes are base64-encoded into a data:image/png;base64,... URL and sent as an image_url content part alongside the text prompt. The model receives the image inline in the same JSON request as the instructions.

Q: Does the model return reliable structure?

Constrain the response to a JSON object and give it an explicit schema, as the example does. The model then returns just the fields you asked for. Treat visual reads of small or low-contrast text with care and validate values that must be exact.

Q: Is this more expensive than DOM scraping?

Yes. An image consumes far more tokens than the equivalent HTML snippet, so vision scraping costs more per page. Use it where it removes a real problem — visual-only data — and parse the DOM where the text is already available.

Q: Can I mix the two approaches?

Yes, and it is often the right design. A single Scraping Browser session can serve both: parse the DOM for the fields that live in clean markup and screenshot the region that does not, so each part of the page is read the cheapest reliable way.