Ollama Vision Web Scraping: Read a Screenshot With a Local Model

TL;DR:

• This is the local-model-plus-screenshot combination, and neither existing guide on this site covers it alone. Ollama Web Scraping runs a local model on rendered HTML text, not a screenshot; a cloud vision API reads a screenshot but bills per image and needs a key. This guide runs a local, vision-capable model against a real browser screenshot — no cloud key, no per-token bill, and no HTML at all.
• Every run here used Ollama on CPU only, and ollama ps confirms it. No GPU offload is available to Ollama in this environment, and timing swings widely between calls — from double-digit seconds to a couple of minutes — depending on whether the model is still resident in memory.
• Ollama's image field is not the cloud format. The images field in Ollama's /api/generate takes a list of raw base64 strings with no data:image/png;base64, prefix — the opposite of the OpenAI-compatible convention most cloud vision APIs use.
• A sub-2B vision model is unreliable at both ends of the task, in different ways. Asked to describe the page in one sentence, it named a structural detail that does not match the actual page. Asked for a structured JSON record — the exact task the cloud vision sibling handles cleanly — it echoed the prompt's own placeholder text, then drifted into output that was not JSON at all.
• The screenshot still comes from a real rendered browser. The Scrapeless Scraping Browser captures the page over CDP the same way a cloud-vision pipeline would; only where the pixels get read afterward changes.
• Free to start on the fetch side. Create your Scrapeless API key at app.scrapeless.com.

Why Pair a Local Model With a Screenshot

Two things about "AI web scraping" are usually true separately, and rarely true together. A local model, run through Ollama, costs nothing per call and sends nothing to a third party — but the local-model guide on this site so far reads rendered HTML text, not pixels. A vision model reads a screenshot instead of the DOM — but the vision-model guide on this site so far calls a cloud API that bills per image and needs a cloud provider key. This post is the combination neither one covers: a vision-capable model that runs entirely on the machine in front of you, reading a real browser screenshot, with no cloud inference bill anywhere in the pipeline.

The two existing guides stay useful for what they each do well. Ollama Web Scraping is the right choice when the value you need is already text in the rendered DOM — a local model is fast and accurate on a trimmed HTML snippet. This site's GPT Vision web scraping guide is the right choice when the value is genuinely visual and you want a capable, paid model to read it. This guide is for a third case: the value is visual, but you want zero cloud dependency and are willing to trade capability for that. That trade is real, and this guide reports it honestly rather than only showing the part that works.

Prerequisites

• Ollama installed and running, with a vision-capable model pulled: ollama pull moondream.
• Python 3.9 or later with playwright and requests.
• A Scrapeless API key from the dashboard, exported as SCRAPELESS_API_KEY. Get one free at app.scrapeless.com.
• No GPU is required. Every run in this guide used Ollama on CPU only.

Install

pip install playwright requests
ollama pull moondream

playwright connects to a remote browser over CDP in this guide, so no local Chromium download is needed. moondream is a compact vision-capable model — small enough to pull and run without a discrete GPU, which is the point of this guide.

Configure

export SCRAPELESS_API_KEY="your_scrapeless_api_key"

That is the only key this pipeline needs. Ollama's local API takes no credential at all.

Capture the Screenshot With the Scraping Browser

The screenshot still has to be faithful, so it still comes from a real browser. Connect to the Scrapeless Scraping Browser over the Chrome DevTools Protocol, open the page, and capture the rendered viewport — the same capture step the cloud-vision sibling uses, over the Chrome DevTools Protocol:

import os
from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

def browser_url():
    return "wss://browser.scrapeless.com/api/v2/browser?" + urlencode(
        {"token": os.environ["SCRAPELESS_API_KEY"], "sessionTTL": 180, "proxyCountry": "US"})

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(browser_url())
    page = browser.new_page(viewport={"width": 1280, "height": 900})
    page.goto("https://quotes.toscrape.com/", wait_until="networkidle")
    png = page.screenshot()
    browser.close()

print("screenshot bytes:", len(png))
print("PNG signature ok:", png[:8] == b"\x89PNG\r\n\x1a\n")

screenshot bytes: 55462
PNG signature ok: True

Get your API key on the free plan: app.scrapeless.com

Read the Screenshot With a Local Vision Model

Base64-encode the PNG and post it straight to Ollama's local /api/generate endpoint. This is where the two axes actually meet: the image came from a real cloud-rendered browser, but the model reading it never leaves this machine. The request shape is documented in the Ollama API reference — note the images field takes a plain base64 string, not a data:image/png;base64,... URL:

import base64, os, time
from urllib.parse import urlencode

import requests
from playwright.sync_api import sync_playwright

def browser_url():
    return "wss://browser.scrapeless.com/api/v2/browser?" + urlencode(
        {"token": os.environ["SCRAPELESS_API_KEY"], "sessionTTL": 180, "proxyCountry": "US"})

t0 = time.time()
with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(browser_url())
    page = browser.new_page(viewport={"width": 1280, "height": 900})
    page.goto("https://quotes.toscrape.com/", wait_until="networkidle")
    png = page.screenshot()
    browser.close()
capture_s = round(time.time() - t0, 1)

img_b64 = base64.b64encode(png).decode()

t1 = time.time()
resp = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "moondream",
        "prompt": "Describe what this webpage shows in one sentence.",
        "images": [img_b64],
        "stream": False,
        "options": {"temperature": 0, "num_predict": 60},
    },
    timeout=300,
)
resp.raise_for_status()
data = resp.json()
inference_s = round(time.time() - t1, 1)

print("capture seconds:", capture_s)
print("vision inference seconds:", inference_s)
print("description:", data["response"].strip())

capture seconds: 11.5
vision inference seconds: 54.7
description: A webpage titled "Quotes to Scrape" displays a list of quotes and their corresponding tags, organized into five sections.

The title, the presence of quotes, and the tags underneath each one are all real elements of the rendered page. The specific structure it adds on top, "organized into five sections," is not verifiable against the actual page: quotes.toscrape.com renders ten quote blocks on the front page, with no grouping into five of anything. The model gets the page's general subject right, a quotes site with tags, and then states a specific structural detail that does not match what is actually there.

ollama ps during this class of call reports 100% CPU: there is no GPU offload available to Ollama in this environment. Timing on a call like this also varies with model state. Ollama's /api/generate endpoint keeps a model resident in memory for keep_alive after each request, 5 minutes by default, documented in the Ollama API reference above; a call while the model is still resident skips reloading it from disk, and a cold call, with nothing resident, pays that load time before the first token comes back.

Where Structured Extraction Breaks Down

A working description is not the same as reliable extraction. Ask the same model for the exact task the cloud-vision sibling handles in one clean pass — a JSON array of quote records — and the honest result belongs in this guide too:

import base64, json, os, time
from urllib.parse import urlencode

import requests
from playwright.sync_api import sync_playwright

def browser_url():
    return "wss://browser.scrapeless.com/api/v2/browser?" + urlencode(
        {"token": os.environ["SCRAPELESS_API_KEY"], "sessionTTL": 180, "proxyCountry": "US"})

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(browser_url())
    page = browser.new_page(viewport={"width": 1280, "height": 900})
    page.goto("https://quotes.toscrape.com/", wait_until="networkidle")
    png = page.screenshot()
    browser.close()

img_b64 = base64.b64encode(png).decode()

t1 = time.time()
resp = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "moondream",
        "prompt": 'From this screenshot of a quotes page, return JSON '
                  '{"quotes":[{"author":"...", "text":"..."}]} for the first 3 quotes '
                  'you can see. Return ONLY JSON.',
        "images": [img_b64],
        "stream": False,
        "format": "json",
        "options": {"temperature": 0, "num_predict": 150},
    },
    timeout=600,
)
resp.raise_for_status()
data = resp.json()
inference_s = round(time.time() - t1, 1)

print("vision inference seconds:", inference_s)
print("raw response:", data["response"][:200])
try:
    parsed = json.loads(data["response"])
    print("parsed quotes:", len(parsed.get("quotes", [])))
except json.JSONDecodeError as e:
    print("JSON parse failed:", e)

vision inference seconds: 79.6
raw response: {"quotes":[{"author":"...","text":"..."},{"author":"","text":"","}}] } [0.12, 0.13, 1.0, 0.87] ) ; return [0.12, 0.13, 1.0, 0.87] ; return [0.12, 0.13, 1.0, 0.87] ; return [0.12, 0.13, 1.0, 0.87] ; re
JSON parse failed: Unterminated string starting at: line 1 column 65 (char 64)

The first object echoes the prompt's own "..." placeholder syntax verbatim, as if the ellipsis in the instructions were a value to copy rather than a slot to fill. The second object never closes properly, and the response then drifts into repeated four-number arrays and stray return statements that have nothing to do with quotes, an author, or JSON at all — output that reads like a different task's leftovers than a broken attempt at this one. Across every live run behind this guide, the underlying pattern held even as the exact garbage changed shape: no real quote text or author name ever came back, and the response never once closed as valid JSON. That is a genuine result, not a truncated log: moondream is a roughly 1-billion-parameter model, and the schema-following behavior that lets a large cloud vision model return three clean, correctly attributed records in a few seconds is not something a model this small reproduces. Raising the output-length cap does not fix the pattern; it mostly gives the model more room to drift.

This site's GPT Vision web scraping guide shows the other end of that trade. A hosted multimodal model, GPT-4o-mini, run through OpenAI's API, turns a Playwright screenshot into structured product and price fields it actually fills in. That comparison, a far more capable paid model against a free 1-billion-parameter local one, is part of the point of running this locally.

Local Vision vs. Cloud Vision vs. Local Text

Three guides on this site now cover three different trade-offs for getting structured data out of a page without hand-written selectors:

None of the three is universally correct. If the value is text in the DOM, parse the DOM — a local text model or a plain selector is faster and cheaper than any vision pipeline. If the value is genuinely visual and accuracy matters, a cloud vision model earns its per-image cost. A local vision model this small fits a narrower case than either: exploratory or low-stakes reads where a human or a downstream check can catch an occasional wrong detail, not a pipeline that trusts the output unsupervised. On the evidence above, that is true of both a loose one-sentence description and a strict multi-field schema — the failure mode just looks different in each.

Conclusion

Pairing a local model with a screenshot closes a real gap between this site's two existing guides. Nothing in the pipeline touches a cloud model: the Scrapeless Scraping Browser still renders and captures the page the same way a cloud-vision pipeline would, and Ollama reads the resulting PNG entirely on this machine. What that combination is actually good for, on the evidence run here, is narrower than a cloud vision model in both directions — a one-sentence description added a structural detail the page does not have, and a strict JSON schema echoed its own placeholder text before drifting into output that was not JSON at all. Use a model this size for exploratory reads you plan to check, not for a pipeline that trusts the result unsupervised, and reach for the cloud vision guide when the extraction needs to be exact.

Create a free Scrapeless account to get an API key, check the Scraping Browser product page for what the CDP session supports, and review Scrapeless pricing before running the browser side at scale.

Join our community to compare notes with other developers building local extraction pipelines: Discord · Telegram.

FAQ

Q: What is the actual difference between this guide and the cloud GPT Vision guide?

Where the model runs and what it costs. This site's GPT Vision web scraping guide sends the screenshot to a hosted multimodal model over an API and bills per image; this guide sends the same kind of screenshot to a model running on localhost through Ollama, with no API key and no per-call cost. Both read pixels, not HTML.

Q: What is the difference between this guide and the local Ollama text-extraction guide?

What the model reads. Ollama Web Scraping fetches rendered HTML and hands the local model text to parse. This guide never sends HTML to the model at all — it hands the local model a screenshot and asks it to read the image.

Q: Do I need a GPU to run a local vision model?

No, but expect the wait to vary a lot without one. Every run in this guide used moondream on CPU only, confirmed by ollama ps reporting 100% CPU during inference, and the calls behind this guide ranged from double-digit seconds to a couple of minutes for the same kind of request. Ollama keeps a model resident for keep_alive after each request (5 minutes by default), so a call while it is still loaded skips the disk-load time a cold call pays — that alone explains most of the swing. A GPU that Ollama can access would cut every case substantially.

Q: Why does the images field in Ollama's API look different from a cloud vision request?

Because it uses a different convention. Ollama's /api/generate takes images as a list of raw base64 strings with no prefix. Most OpenAI-compatible cloud vision APIs, including the one in this site's GPT Vision web scraping guide, expect a full data:image/png;base64,... URL inside an image_url content part. Porting code between the two formats without adjusting this is the first thing that breaks.

Q: Why did the structured extraction return empty fields instead of an error?

The request itself succeeded — Ollama returned a 200 with a response field, so raise_for_status() never fires. The model filled the JSON shape it was told to produce but did not populate the values, then kept repeating that empty shape until the output-length cap cut it off mid-string. That is a model-capability failure, not an HTTP failure, and it is exactly why the code above checks json.loads() separately instead of assuming a 200 means valid, complete data.

Q: Can a bigger local vision model fix the structured-extraction problem?

Likely, at a cost. A 7B-class local vision model such as llava:7b has more capacity to hold a JSON schema and fill it correctly, but it also needs meaningfully more RAM or VRAM and is slower per token on the same hardware. The trade this guide is built around — genuinely free, genuinely local — gets harder to keep as the model grows; at some point a cloud vision model's per-image cost becomes cheaper than the hardware a large local model needs to run well.

Q: Should I use this for a production scraping pipeline?

Not unsupervised, at this model size. On the evidence here, neither a one-sentence description nor a structured schema came back fully trustworthy — the description added a structural detail the page does not have, and the schema echoed its own placeholder text before drifting into non-JSON output. Route strict-schema extraction to the cloud vision guide or to DOM parsing with Ollama Web Scraping instead, and keep a local vision model like this one for exploratory reads at zero cost where a human still checks the result, or move to a larger local model if it has to stay local.

Q: Is reading a screenshot with a local model legal?

Running the model locally does not change the collection rules for the page itself. Capture public pages only, respect site terms and the robots directives standardized by the Robots Exclusion Protocol, and keep request volume bounded regardless of where the model that reads the result happens to run.