Hugging Face smolagents + Scrapeless MCP: Build an AI Web Scraper in Python

TL;DR:

• A Hugging Face agent gets 21 live web tools from one MCP endpoint. Pointing ToolCollection.from_mcp at https://api.scrapeless.com/mcp hands a smolagents CodeAgent browser control, page scraping, and Google Search and Trends, while rendering, proxy routing, and anti-detection stay server-side.
• The hosted path is pure Python. Streamable HTTP with an x-api-token header replaces any local server process — no Node.js, one pip install "smolagents[mcp]".
• The tool surface works before a model does. A plain function call to scrape_markdown returns a live page as clean markdown — 4,249 characters for the demo page in this guide — so you can test the wiring with only a Scrapeless API key.
• An AI scraper extracts by meaning, not by selector. The agent reads the markdown and returns the fields you ask for, so a site redesign that would break a CSS-selector script usually costs you nothing.
• The one extra prerequisite is a model key. Tool listing, direct tool calls, and agent construction all run without one; only the reasoning round-trip needs a Hugging Face token or another supported provider.
• Free to start. Create your API key on the free plan at app.scrapeless.com.

What this integration enables

A selector-based scraper is a bet that the target page never changes, and that bet loses often. An AI scraper takes a different position: fetch the page as clean text, let a language model pull out the fields you want, and stop caring which div the price lives in this week.

smolagents is Hugging Face's small agent library — its agents write Python to call tools instead of emitting JSON tool calls. What it lacks on its own is a way to reach the live web. That is the job of the Model Context Protocol: the Model Context Protocol specification defines how a server advertises typed tools that any client can list and invoke. If the protocol itself is new to you, the primer on what MCP is and how it works covers the concept end to end.

Wire the two together and you get a programmatic AI scraper in a few dozen lines of Python: smolagents supplies the reasoning loop, the Scrapeless MCP server supplies fetching, rendering, and search as callable tools. This guide builds that scraper step by step and shows exactly which parts run with nothing but a Scrapeless key.

Why Scrapeless MCP

The Scrapeless MCP server exposes the scraping infrastructure as 21 typed tools, and the heavy work happens on the server, not in your process. scrape_html, scrape_markdown, and scrape_screenshot capture single pages in different shapes. Sixteen browser_* tools operate cloud browser sessions on the Scraping Browser — an anti-detection cloud browser powered by self-developed Chromium — for jobs where an agent must click, type, and scroll. google_search and google_trends cover discovery.

Three properties matter for this build:

• One key, hosted transport. The same Scrapeless API key that drives the rest of the platform authenticates the MCP endpoint. Your Python process never launches a browser or a Node server.
• Model-free testability. Tools list and execute without any LLM in the loop, so the integration can be proven layer by layer instead of debugged through an agent's reasoning.
• A markdown-first extraction path. The markdown capture of a page is a fraction of the size of its raw HTML, which means fewer tokens per extraction and less noise for the model to read through. scrape_markdown returns exactly that.

The same endpoint also plugs into LangChain if that is your stack — the LangChain + Scrapeless MCP guide walks the same surface from the adapter side.

Prerequisites

• Python 3.10 or newer — the runs in this guide used Python 3.12.
• A Scrapeless API key from the dashboard — the developer docs cover key creation and the endpoint reference.
• For the final agent round-trip only: a Hugging Face token (or credentials for any model provider smolagents supports). Every step before that runs without it.

Install and configure

One package with one extra brings in the agent library and the MCP client plumbing. These versions are the ones this guide was written against — smolagents 1.26.0, mcp 1.27.1, mcpadapt 0.1.20:

pip install "smolagents[mcp]==1.26.0"

Export your key so the scripts can read it from the environment rather than from source code:

export SCRAPELESS_API_KEY="sk_your_key_here"

Connect over streamable HTTP and list the tools

The connection is a dictionary, not a config file. ToolCollection.from_mcp accepts the same parameters as the underlying streamable HTTP client, so the endpoint URL, the transport name, and the auth header travel in one literal:

# connect_and_list.py — handshake with the Scrapeless MCP server, list the tools
import os

from smolagents import ToolCollection

server = {
    "url": "https://api.scrapeless.com/mcp",
    "transport": "streamable-http",
    "headers": {"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
}

with ToolCollection.from_mcp(server, trust_remote_code=True, structured_output=False) as tc:
    names = sorted(tool.name for tool in tc.tools)
    print(f"Tool count: {len(names)}")
    print("\n".join(names))

The context manager owns the connection lifecycle: it performs the MCP handshake on entry and disconnects cleanly on exit. One parameter deserves a word: the MCP tools specification lets a server return results as plain text or as structured content, and the Scrapeless tools return text — so pass structured_output=False explicitly. smolagents 1.26 warns whenever the parameter is omitted, because its default is scheduled to flip in a future release.

A correct handshake prints Tool count: 21 followed by the names: sixteen browser_* tools, google_search, google_trends, scrape_html, scrape_markdown, and scrape_screenshot.

A stdio route also exists, for clients that prefer to launch a local server process — the same 21 tools behind a different lifecycle:

{
  "mcpServers": {
    "scrapeless": {
      "command": "npx",
      "args": ["-y", "scrapeless-mcp-server"],
      "env": { "SCRAPELESS_KEY": "sk_your_key_here" }
    }
  }
}

For a Python-only build, streamable HTTP is the shorter path: nothing to install beyond pip, nothing to keep running.

Get your API key on the free plan: app.scrapeless.com

Call scrape_markdown before any model is involved

Every tool in the collection is a callable smolagents Tool object, so the scraping layer can be exercised directly — no agent or model key involved:

# call_tool.py — execute one MCP tool as a plain function call
import json
import os

from smolagents import ToolCollection

server = {
    "url": "https://api.scrapeless.com/mcp",
    "transport": "streamable-http",
    "headers": {"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
}

with ToolCollection.from_mcp(server, trust_remote_code=True, structured_output=False) as tc:
    tools = {tool.name: tool for tool in tc.tools}
    raw = str(tools["scrape_markdown"](url="https://quotes.toscrape.com/"))
    # The hosted tool returns the page as a JSON-quoted string after a
    # "Response:" line — decode it to get the markdown itself.
    body = raw.split("\n\n", 1)[1] if raw.startswith("Response:") else raw
    text = json.loads(body) if body.startswith('"') else body
    print(f"scrape_markdown returned {len(text):,} characters of markdown")
    print(text[:160])

Against the quotes demo site this returns 4,249 characters of markdown, beginning with the page title and the first quote — readable text with the page chrome already stripped. That single call is the entire fetch layer of the scraper. Everything after it is interpretation.

Attach the tools to a CodeAgent

Binding the collection to an agent is one constructor, and it works before any model call happens. The agent object indexes every MCP tool by name next to its built-in final_answer:

# attach_agent.py — hand the MCP tool surface to a smolagents CodeAgent
import os

from smolagents import CodeAgent, InferenceClientModel, ToolCollection

server = {
    "url": "https://api.scrapeless.com/mcp",
    "transport": "streamable-http",
    "headers": {"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
}

with ToolCollection.from_mcp(server, trust_remote_code=True, structured_output=False) as tc:
    model = InferenceClientModel(model_id="Qwen/Qwen2.5-72B-Instruct")
    agent = CodeAgent(tools=[*tc.tools], model=model, add_base_tools=False)
    print(sorted(agent.tools.keys()))

add_base_tools=False keeps the toolbox to the MCP tools plus the agent's built-in final_answer. For a scraper that should fetch pages and nothing else, you can narrow it further — pass only the tool you want, as in tools=[t for t in tc.tools if t.name == "scrape_markdown"] — and the model physically cannot wander into browser sessions or search calls it does not need. A smaller toolbox also means a smaller system prompt and fewer wrong turns from the model.

Prompt-driven use: the AI scraper run

The extraction step is a prompt, not a parser. You tell the agent which page to read and which fields to return, and the agent decides to call scrape_markdown, reads the result, and assembles the answer — smolagents describes each tool to the model with typed inputs, the same structure the JSON Schema specification defines for machine-readable field constraints.

Note: This final step is the one prerequisite gap in this guide — the agent round-trip needs a model provider. Set HF_TOKEN with a Hugging Face token (or configure another provider smolagents supports) before running it. Every block above runs with only the Scrapeless key.

# run_scraper.py — the model round-trip (requires HF_TOKEN or another provider)
result = agent.run(
    "Call scrape_markdown on https://quotes.toscrape.com/ and return a JSON array "
    "of the quotes on the page. Each item must have exactly these keys: "
    "text (string), author (string), tags (array of strings). "
    "Return only the JSON array, no commentary."
)
print(result)

Prompt shape controls output shape. Naming the exact keys and types, demanding "only the JSON array", and keeping one page per run gives you output you can json.loads and validate downstream. When a field is missing on the page, instruct the agent to use null rather than invent a value — models fill gaps confidently unless told not to.

What you get back

From the fetch layer, you get markdown as a string: the page title as a heading, link text preserved in brackets, body text in reading order. The 4,249-character capture from the quotes site starts like this:

# [Quotes to Scrape](https://quotes.toscrape.com/)

[Login](https://quotes.toscrape.com/login)

“The world as we have created it is a process of our thinking.

From the agent run, you get whatever contract your prompt enforced — here, a JSON array of {text, author, tags} objects, one per quote on the page. The value of the arrangement shows up on the day the target site changes its class names: the markdown still contains the quotes, the prompt still names the fields, and the scraper still returns the same schema while a selector-based script returns nothing.

Conclusion

The integration is three small moves: point ToolCollection.from_mcp at the hosted endpoint, prove the fetch layer with a direct scrape_markdown call, then bind the tools to a CodeAgent and let a prompt do the extraction. Each layer is testable on its own, only the last one needs a model key, and the part most likely to break in a classic scraper — the parsing — is the part the model absorbs.

Ready to Give Your Agent a Real Web Surface?

The MCP endpoint authenticates with the same API key as the rest of the Scrapeless platform — plans and included volumes are on the pricing page. Create a key on the free plan at app.scrapeless.com and the handshake script above will print your 21 tools in under a minute.

FAQ

Q: What is an AI scraper?

An AI scraper is a scraper that uses a language model for the extraction step instead of hand-written parsing rules. A conventional scraper couples fetching and parsing to a specific page structure; an AI scraper fetches the page as text and asks a model to return named fields, which keeps working across layout changes that would break selectors.

Q: Do I need a Hugging Face token to call the Scrapeless tools?

No. Listing tools, calling scrape_markdown directly, and constructing the CodeAgent all authenticate with only the Scrapeless API key. The Hugging Face token (or another provider's key) is needed for exactly one thing: the agent.run() reasoning round-trip.

Q: Should I connect over streamable HTTP or stdio?

Use streamable HTTP for Python projects: it needs no local process and authenticates with a header. The stdio transport (npx -y scrapeless-mcp-server, authenticated through the SCRAPELESS_KEY environment variable) suits desktop MCP clients that manage server processes themselves. Both transports expose the same tool surface.

Q: Can the agent use just one tool instead of all 21?

Yes. Filter the collection before constructing the agent — tools=[t for t in tc.tools if t.name == "scrape_markdown"] — and the model only ever sees that tool. For single-purpose scrapers this is the recommended shape: the system prompt shrinks and the model cannot start browser sessions you never intended.

Q: What about JavaScript-heavy pages or pages behind anti-bot challenges?

Rendering happens server-side, so your Python code does not change. scrape_html and scrape_markdown handle pages that need JavaScript execution, and the browser_* tools drive full cloud browser sessions for flows that require clicking or typing. Proxy routing and anti-detection are part of the managed service rather than something the agent must reason about.

Q: Which models work with smolagents?

Any provider the library supports. InferenceClientModel covers models served through Hugging Face inference providers, and the library also ships OpenAIModel, AzureOpenAIModel, AmazonBedrockModel, LiteLLMModel, and local backends such as TransformersModel — see the smolagents model reference for the current list. The MCP side is model-agnostic: tools look identical whichever model reasons over them.

Q: Is it legal to scrape with an AI agent?

The same rules apply as for any scraper: collect public pages only, respect the target site's terms and robots directives, keep volumes bounded, and handle any personal data under the privacy laws that apply to you. An agent changes how extraction happens, not what you are allowed to collect — when in doubt, ask counsel before you scale a workload.