The Complete Guide to Web Scraping in Zencoder with Scrapeless MCP Server

Key Takeaways:

• One MCP server, every Zencoder agent. Zencoder runs Coding Agent, Unit Testing Agent, Ask Agent, E2E Testing Agent, Repo Info Agent, and Web Dev Agent inside both the VS Code extension and the JetBrains plugin. Adding the Scrapeless MCP Server to either client gives every one of those agents a hardened cloud browser, residential proxies in 195+ countries, an SERP scraper, and one-shot Markdown/Screenshot helpers — through a single config block.
• The agent stops getting blocked. Zencoder ships powerful Repo Grokking and codegen, but its default web fetch hits the same JavaScript shells and anti-bot gates any anonymous HTTP request would. Routing through Scrapeless Scraping Browser swaps in cloud-side JavaScript rendering, residential-proxy egress, and anti-detection fingerprinting on every session.
• 20 MCP tools, one namespace. Browser primitives (browser_create, browser_goto, browser_get_html, browser_get_text, browser_snapshot, browser_click, browser_type, browser_press_key, browser_scroll, browser_screenshot, browser_close), one-shot page helpers (scrape_html, scrape_markdown, scrape_screenshot), and Google data tools (google_search, google_trends). The agent's planner composes them per turn.
• Two config schemas, one server. VS Code reads "zencoder.mcpServers" from settings.json and currently supports stdio only. JetBrains 2.13+ uses a bare object schema and supports stdio, streamable-HTTP, and OAuth2 transports. The same scrapeless server entry drops into both — only the wrapper key changes.
• Skills and MCP compose. Zencoder's Skills system (~/.zencoder/skills/<name>/SKILL.md, YAML front matter + markdown) teaches an agent how to use a tool; MCP defines what tools exist. They are additive — install both and the agent gets the tool surface plus the playbook for using it.
• Model-agnostic by design. Zencoder routes between Haiku 4.5, Sonnet 4.6, Opus 4.6, Opus 4.7, Gemini 3 Pro, Gemini 3 Flash, GPT-5.4, and GPT-5.2 through the same agent loop. The Scrapeless MCP integration runs identically across all of them — the protocol is the contract.
• Free to start. New Scrapeless accounts include free Scraping Browser runtime — sign up at app.scrapeless.com. Full pricing at scrapeless.com/en/pricing.

Introduction: the agent is only as useful as its access to the live web

Zencoder is an AI coding orchestration platform built around a multi-agent workflow — Coding Agent for implementation, Repo Info Agent for codebase context, Unit Testing Agent for coverage, E2E Testing Agent for browser tests, Ask Agent for explanations, and Web Dev Agent for UI work. It ships in two surfaces: a Visual Studio Code extension and a JetBrains plugin. For local code, the picture is complete — Zencoder reads the repo, plans the change, and writes the diff.

The moment a task needs current, public web data, the picture gets thinner. Zencoder's default web fetch returns the same JavaScript shell that any anonymous HTTP request would, and on commercial sites the rendered DOM lives behind anti-bot challenges, residential-proxy-gated SERPs, and JavaScript-only single-page apps that no headless fetch reaches without help. A prompt like "open this competitor pricing page and pull the plan grid" stops being deterministic once the target site sits behind Cloudflare Turnstile.

This post wires the Scrapeless MCP Server into Zencoder through a single config block. After the wire-up, every Zencoder agent has 20 MCP tools mapped to a hardened cloud browser, a Google Search scraper, a Google Trends scraper, and one-shot HTML/Markdown/Screenshot helpers. The agent picks which tool to call per turn; the cloud browser handles JavaScript rendering, residential-proxy egress, and anti-detection fingerprinting; the IDE keeps owning the codegen, the file tree, and the terminal. For the same Scrapeless surface in Google Antigravity, see the Antigravity integration walkthrough; for the canonical MCP server reference, see the MCP server walkthrough for Google Maps.

What You Can Do With It

• Live API and library research inside the IDE. Ask the Coding Agent to fetch the current docs page for a library through scrape_markdown and reason against the rendered text rather than against a training-cutoff memory of the API.
• Competitor pricing snapshots in the workspace. Drop competitor URLs into the agent chat; get back structured pricing rows the agent can paste into a comparison file alongside the code that consumes them.
• Public-dataset ingestion into the project being built. Have the agent walk a directory listing, extract typed records, and write them straight into a JSON fixture, a SQL seed, or a TypeScript type next to the model layer.
• Region-aware SERP and trend signals. Use google_search and google_trends from inside the agent loop to seed feature flags, A/B copy, or content templates with locale-specific evidence.
• Markdown corpora for RAG and project docs. scrape_markdown returns publisher pages as clean markdown the agent can chunk, embed, or store as project-local context.
• Visual references for design and UX work. scrape_screenshot and browser_screenshot capture rendered pages for visual diff, layout reference, or design inputs the Web Dev Agent can attach to the conversation.
• Multi-agent runs from one prompt. Repo Info Agent gathers codebase context, Coding Agent executes the browser_* chain, Unit Testing Agent verifies the resulting fixture shape — all inside one Zencoder turn.

At Scrapeless, we only access publicly available data while strictly complying with applicable laws, regulations, and website privacy policies. The content in this post is for demonstration purposes only.

Why Scrapeless MCP Server

Scrapeless MCP Server is the protocol-level surface in front of Scrapeless Scraping Browser — a customizable, anti-detection cloud browser designed for AI agents — plus the Scrapeless data tools (Google Search, Google Trends, page-level scrape helpers). For Zencoder specifically, it brings:

• Cloud-side JavaScript rendering. The cloud browser is full Chromium with the page hydrated before extraction, so SPAs, infinite-scroll feeds, and lazy-loaded panels are first-class targets for browser_goto + browser_get_html.
• Residential proxies in 195+ countries. Geo-bound queries return the listings a local user would see. The proxy region is set on the Scrapeless account; per-query region selection is exposed through the scrapeless-scraping-browser CLI for workflows that need it.
• Anti-detection fingerprinting on every session. UA, timezone, language, screen resolution, WebGL, and canvas randomized per session, so long scroll sessions render identically to organic traffic.
• Session persistence. The browser_create task id is reused across subsequent browser_* calls in the same agent turn; cookies, scroll position, and navigation history all live inside one cloud-browser session.
• One MCP namespace. Every operation the agent needs is a single tool call — discoverable from Zencoder's Agent Tools panel, schema-validated by the protocol, and matched to prompts by description.
• Free runtime on new accounts. Get an API key on the free plan at app.scrapeless.com. The full MCP tool surface is documented at github.com/scrapeless-ai/scrapeless-mcp-server and the platform docs live at docs.scrapeless.com.

Prerequisites

• Zencoder installed. Either the VS Code extension or the JetBrains plugin (full stdio + streamable-HTTP + OAuth2 MCP support from JetBrains 2.13+).
• A Scrapeless account and API key — sign up at app.scrapeless.com and copy the key from Settings → API Key Management.
• Node.js 18 or newer on the workstation. Zencoder spawns npx scrapeless-mcp-server as a child process for stdio mode.
• Basic familiarity with editing JSON — the Zencoder MCP config lives in settings.json.

Install

The setup is five sub-steps. The JSON schema differs between VS Code and JetBrains — pick the section that matches the IDE you're configuring.

1. Get your Scrapeless API key

Sign up at app.scrapeless.com, open the dashboard, and from Settings → API Key Management create a key. Copy the value — it goes into the MCP config in step 3.

2. Open Zencoder's MCP config

Option A — Agent Tools UI (suggested for both IDEs):

• Click the … more-options menu at the top right of the Zencoder chat panel.
• Select Agent Tools.
• Open the Custom tab and click Add Custom MCP.
• Fill in the configuration form (name, command, args, env). The UI writes to settings.json for you.

Option B — Edit settings.json directly:

• VS Code: Open the … menu in the Zencoder chat → Settings → scroll to the MCP Servers section → click Edit in settings.json.
• JetBrains: Open File → Settings (or JetBrains IDE → Settings on macOS) → expand Tools → Zencoder → MCP Servers.

3. Add the Scrapeless MCP server

The JSON schema is different across the two IDEs — VS Code wraps the entries inside a "zencoder.mcpServers" top-level key; JetBrains uses a bare object.

VS Code — stdio mode (currently the only transport supported on VS Code):

"zencoder.mcpServers": {
  "scrapeless": {
    "command": "npx",
    "args": ["-y", "scrapeless-mcp-server"],
    "env": {
      "SCRAPELESS_KEY": "YOUR_SCRAPELESS_KEY"
    }
  }
}

JetBrains — stdio mode:

{
  "scrapeless": {
    "command": "npx",
    "args": ["-y", "scrapeless-mcp-server"],
    "env": {
      "SCRAPELESS_KEY": "YOUR_SCRAPELESS_KEY"
    }
  }
}

Save the file. On first run, npx -y scrapeless-mcp-server downloads the package; subsequent restarts reuse the cached version.

4. Or use HTTP streamable mode (JetBrains 2.13+ only)

JetBrains 2.13+ supports stdio, streamable-HTTP, and OAuth2 MCP transports; VS Code is currently stdio-only (streamable-HTTP and OAuth2 are tracked as "coming soon" in the Zencoder docs). For a remote agent that runs in a hosted dev container or CI sandbox where npx cannot reliably spawn a long-lived child process, JetBrains users can point Zencoder at the Scrapeless-hosted MCP endpoint:

{
  "scrapeless": {
    "type": "streamable-http",
    "url": "https://api.scrapeless.com/mcp",
    "headers": {
      "x-api-token": "YOUR_SCRAPELESS_KEY"
    }
  }
}

The same YOUR_SCRAPELESS_KEY works in both modes. VS Code users stay on the stdio block in step 3 until Zencoder ships streamable-HTTP for VS Code.

5. Verify the connection

Apply the changes — JetBrains requires reopening the project after editing settings; VS Code typically picks the change up on reload. Make sure the Coding Agent is enabled. In a fresh Zencoder chat session, type:

"Open https://example.com with the Scrapeless browser tool and tell me the page title."

The agent should call browser_create, then browser_goto, then browser_get_text (or browser_get_html), and reply with "Example Domain". If that returns clean, the MCP server is wired, the API key is valid, and the cloud browser is reachable.

If it does not, the common first-run failures and fixes are:

How you actually use this: prompt your Zencoder agent

After install, the agent owns the discover → extract loop. The MCP server hands Zencoder a typed tool list; the agent reads the tool descriptions and composes calls turn by turn based on the prompt. There is no tools.json to author and no manual MCP call to issue from the IDE side.

Zencoder reserves @ shortcuts for the built-in Native Integrations only — @Jira works because Jira is a native integration; MCP servers are not addressed with @<servername>. The agent's planner matches the prompt against the tool descriptions, so phrase the request in terms of the action ("render the page at…", "search Google for…", "scrape that URL as markdown") and the right Scrapeless tool gets called.

Prompts you can paste

Worked example: pricing-page extraction inside Zencoder

You type:

"Use the Scrapeless browser tool to open https://example-saas.com/pricing, scroll the plan grid to the bottom, and return name, price, and feature bullets per plan as JSON. Save the result to pricing.json in the open workspace."

The agent's plan (in plain English):

1. Call browser_create to mint a cloud-browser session.
2. Call browser_goto with the pricing URL.
3. Call browser_wait_for against the plan-card landmark so extraction runs against the hydrated DOM rather than the SPA shell.
4. Call browser_scroll to surface any below-the-fold tiers, then browser_get_html.
5. Parse the plan cards into a JSON array; treat any field absent on a card as null rather than failing the extraction.
6. Call browser_close with the sessionId returned by browser_create.
7. Write the array to pricing.json using Zencoder's built-in file tool.

What you get back (illustrative shape):

[
  {
    "name": "Starter",
    "price": "$0 / month",
    "features": ["1 seat", "1,000 events/mo", "Community support"]
  },
  {
    "name": "Pro",
    "price": "$29 / month",
    "features": ["10 seats", "100K events/mo", "Email support", "Custom domains"]
  },
  {
    "name": "Business",
    "price": "Contact sales",
    "features": ["Unlimited seats", "Custom event volume", "SLA", "SSO/SAML"]
  }
]
// Schema reflects what the agent emits when prompted to extract a pricing grid.
// Field values are illustrative samples.

Zencoder drops pricing.json into the workspace tree and surfaces each MCP tool call in the conversation trace so the per-step flow is auditable.

Shaping prompts: control what comes back

Get your API key on the free plan: app.scrapeless.com

The Scrapeless MCP tool surface

The full tool list Zencoder discovers when the scrapeless server connects:

browser_* tools share state via the sessionId returned by browser_create. scrape_* and the Google tools are stateless and route through the Scrapeless API directly without minting a session. Tool arguments use camelCase across the surface (sessionId, proxyCountry).

Five workflows that pay for the integration

Each block below is one prompt-driven workflow inside Zencoder. The Repo Info Agent gathers project context, the Coding Agent runs the MCP chain, and the Unit Testing Agent verifies the resulting fixture shape — all inside one chat turn.

1. Live SERP and trend signals for feature work

A feature flag's copy often depends on what users are currently searching for. Ask the agent to call google_trends for the topic in the target region, then google_search for the head terms, then write the result into a fixture file the running app consumes. The whole loop runs inside one Zencoder turn and refreshes whenever the prompt is reissued.

2. Competitor pricing tracker in a comparison file

Drop a list of competitor pricing URLs into the chat and ask the agent to extract plan name, price, and bullet features per page, dedupe by plan, and write the merged table to competitors.md in the workspace. Subsequent runs diff against the previous snapshot and only update the rows that changed.

3. Public-directory lead lists into the project

For lead-generation prototypes the Coding Agent walks a public directory (a list of software vendors, local businesses, or licensed professionals), calls browser_goto per detail page, extracts contact rows, and writes the records as a seed.sql or leads.json next to the model layer. Residential-proxy egress and JS rendering happen at the cloud-browser layer.

4. RAG corpus from publisher pages

For an embedding pipeline being built inside Zencoder, the agent walks a seed URL list, calls scrape_markdown per page, splits the markdown into chunks, and writes each chunk + URL pair to corpus.jsonl. Because the markdown is already clean, chunk boundaries land on headings and paragraphs rather than mid-tag.

5. Visual regression assets for the Web Dev Agent

For a UI refactor, ask the Web Dev Agent to take a full-page scrape_screenshot of staging and production for every page in routes.ts, then diff the pairs. The screenshots come back as files in the workspace; the diff lives in the conversation.

Skills are separate from MCP — and they compose

Zencoder ships a Skills system alongside MCP. A Skill lives at ~/.zencoder/skills/<name>/SKILL.md with YAML front matter (name, description) and a markdown body that teaches the agent when and how to use a tool. Skills can ship scripts and resources alongside the SKILL.md file.

MCP defines what tools exist; Skills define how to use them well. The two are additive — installing a Skill that teaches the agent to compose browser_create → browser_goto → browser_wait_for → browser_get_html into a clean discover → extract pattern raises the success rate of natural-language prompts against the same MCP tool surface. The blog does not assume any pre-built Scrapeless Skill ships in the Zencoder ecosystem; users who want one can author one at ~/.zencoder/skills/scrapeless/SKILL.md against the tool list above.

What to know before going to production

• Stdout discipline matters. The stdio MCP transport uses stdout for JSON-RPC. A server that writes non-JSON text to stdout breaks the handshake. The current scrapeless-mcp-server build writes logs to stderr; if a custom fork is in use, audit the log destination.
• Per-call region control is not on the MCP surface. The cloud browser routes through the Scrapeless account's configured region. Workflows that need per-query region pinning (US results vs DE vs JP) call the scrapeless-scraping-browser CLI with --proxy-country or keep multiple API keys for different default regions.
• VS Code is stdio-only today. Streamable-HTTP and OAuth2 transports are tracked as "coming soon" for VS Code in the Zencoder docs. JetBrains 2.13+ has all three. A cross-IDE config pattern that needs HTTP must stay on JetBrains or wait for the VS Code update.
• JetBrains requires a project reopen after editing settings.json; VS Code typically picks the change up on reload.
• Concurrency. Keep ≤ 3 concurrent sessions per host for stable runs. For batch jobs that need higher fan-out, drive the CLI from a worker pool instead of issuing MCP calls in parallel from a single agent.
• MCP response shape. Tool responses come back as content[0].text (plain text). The text-returning stateless data tools (google_search, google_trends, scrape_html, scrape_markdown) prefix their payload with Response:\n\n before the body; Zencoder's planner handles the prefix automatically, but custom scripts parsing the raw response need to strip it. scrape_screenshot returns image binary directly. The browser_* tools return their text payloads without the prefix.
• Scrapeless is installed direct from the package, not via the Zencoder MCP Library. As of writing, Scrapeless is not listed in the in-IDE MCP Library catalog — paste the JSON block from step 3 instead of searching the Library. The MCP Library route is also available for the prebuilt MCPs that Zencoder ships with one-click setup.

Conclusion: scale your Zencoder agents' reach across the public web

Wiring Scrapeless MCP Server into Zencoder collapses to one JSON block — "zencoder.mcpServers" in settings.json on VS Code, a bare object on JetBrains — and a reload. After that, every Zencoder agent has a hardened cloud browser, residential-proxy egress, a Google SERP scraper, a Trends scraper, and one-shot HTML/Markdown/Screenshot helpers — reachable through natural-language prompts. The IDE keeps owning the codegen, the file tree, and the terminal; the cloud browser keeps the agent on real, rendered pages.

For the same Scrapeless surface in Google Antigravity, see the Antigravity integration walkthrough. For the canonical MCP server tool surface and a Google Maps worked example, see the MCP server walkthrough. For agent surfaces that prefer the raw CDP path over MCP, see the Hermes integration post. The new-account tier on Scrapeless includes free Scraping Browser runtime; the upgrade paths after the free runtime is consumed are at scrapeless.com/en/pricing. The pattern stays consistent across every agent surface: pin the right transport for the host IDE, snapshot or render before extracting, persist the session across multi-step turns, and treat absent fields as nullable.

Ready to Build Your AI-Powered Data Pipeline?

Join our community to claim a free plan and connect with developers building Zencoder-driven data pipelines on Scrapeless: Discord · Telegram.

Sign up at app.scrapeless.com for free Scraping Browser runtime, and adapt the patterns above to the workspaces, regions, and pages your Zencoder agents need.

FAQ

Q1: Which Zencoder surfaces support the Scrapeless MCP server?
Both the VS Code extension and the JetBrains plugin. The JSON schema differs (VS Code wraps entries in "zencoder.mcpServers"; JetBrains uses a bare object), and JetBrains 2.13+ supports stdio + streamable-HTTP + OAuth2 transports while VS Code is currently stdio-only.

Q2: Is scraping with Zencoder + Scrapeless legal?
Scraping publicly visible data is broadly permitted in most jurisdictions, but rules vary by country and by site terms of service. Review the target site's ToS, respect robots.txt where applicable, do not collect personal data without a lawful basis, and consult counsel for commercial-scale pipelines.

Q3: Do I need a residential proxy?
Yes for any site with meaningful anti-bot protection — most retailers, marketplaces, SERPs, and large SaaS landing pages. The Scrapeless MCP server routes every browser_* and scrape_* call through the residential pool by default; no separate proxy configuration is needed for the call to work.

Q4: Can I pick a residential-proxy region per request from the MCP surface?
Not from the MCP browser_create tool — the cloud-browser session uses the proxy region configured on the Scrapeless account. For per-query region selection (US vs DE vs JP), drive the cloud browser through the scrapeless-scraping-browser CLI with --proxy-country, or maintain multiple API keys for different default regions.

Q5: stdio mode vs streamable-HTTP mode — which one belongs in Zencoder?
VS Code is stdio-only today; JetBrains 2.13+ supports both. Stdio is the right default for a developer workstation — Zencoder spawns npx scrapeless-mcp-server as a child process and talks JSON-RPC over its stdin/stdout. Streamable-HTTP (JetBrains only) is the right default when the host runs in a hosted dev container or CI sandbox where the host cannot keep a long-lived child process alive. Both modes use the same API key.

Q6: VS Code uses "zencoder.mcpServers" but JetBrains uses a bare object — why the difference?
That is how Zencoder's two settings panes are wired. VS Code's settings.json is the IDE's global settings file with extension-namespaced keys, so Zencoder uses "zencoder.mcpServers". JetBrains exposes the MCP Servers section directly under Tools → Zencoder → MCP Servers, so the JSON the user pastes is just the inner object. The command/args/env/type/url/headers keys themselves are identical across both IDEs — only the wrapper changes.

Q7: How do I keep the API key out of the JSON file?
Use a placeholder in the JSON and inject the value through an OS environment variable. Set SCRAPELESS_KEY in the shell that launches Zencoder (export SCRAPELESS_KEY=... on macOS/Linux or [Environment]::SetEnvironmentVariable("SCRAPELESS_KEY", "...", "User") on Windows PowerShell), then reference ${SCRAPELESS_KEY} in the env block of the config. Restart Zencoder so the new variable propagates.

Q8: The first browser_create call returns os error 10054 or 503. What now?
Both are transient session-spin-up errors on the Scrapeless residential-proxy pool. A single retry usually succeeds — ask the agent to call browser_close (if a session was minted) and browser_create again, or wrap the call in a 2–3 attempt retry loop.

Q9: How does Zencoder decide which Scrapeless tool to call?
Each MCP tool ships a name and a schema; Zencoder's planner reads the descriptions and matches them to the prompt. There is no @<servername> shortcut for MCP servers — @Jira and @GitHub route to Native Integrations, not to MCP. Phrase the task naturally ("scrape this URL as markdown", "search Google for X") and the right Scrapeless tool gets selected.

Q10: Are Skills required, or just nice-to-have?
Optional. The MCP server alone gives the agent every Scrapeless tool. A Skill at ~/.zencoder/skills/scrapeless/SKILL.md raises the prompt-to-correct-tool success rate by teaching the agent the discover → extract pattern, but the integration works end-to-end without one.

Q11: Can I run this without Zencoder?
Yes. The Scrapeless MCP Server is a standard MCP server, callable from Claude Desktop, Claude Code, Cursor, OpenAI Codex CLI, Gemini CLI, VS Code + GitHub Copilot Chat, Google Antigravity, or any custom client built against the MCP TypeScript SDK. The same JSON snippet drops into each (with the wrapper key adjusted per client). For non-MCP surfaces, the cloud browser is also reachable through the scrapeless-scraping-browser CLI and the WSS endpoint at wss://browser.scrapeless.com/browser.