AutoGen + Scrapeless: Give Your Agents Live Web Tools via MCP

TL;DR:

• AutoGen loads the Scrapeless MCP Server's tools with mcp_server_tools and hands all 21 to an agent, from scrape_markdown to a full browser set.
• The connection is one object, StreamableHttpServerParams, carrying the endpoint and the x-api-token header; no model client is needed to load or call the tools.
• Each tool arrives as a StreamableHttpMcpToolAdapter, so you can call one directly with run_json({...}, CancellationToken()) before wiring it to an agent.
• Only AssistantAgent.run needs a model-provider key; loading the tools and calling scrape_markdown do not.
• One scrape_markdown call returns the target page as Markdown, ready for the agent to reason over.
• Start on the Scrapeless free plan and give your AutoGen agents real web tools.

AutoGen is Microsoft's framework for multi-agent applications, and its agents are only as capable as the tools they hold. Out of the box, none of those tools reach the live web. The Model Context Protocol closes that gap: point AutoGen at an MCP server and every tool it exposes becomes an AutoGen tool your agent can call, with the same interface as any function tool.

This guide connects AutoGen to the Scrapeless MCP Server, loads its 21 tools, calls one for real, and attaches the set to an AssistantAgent — all verified against the live server. The only step that needs a model-provider key is the agent's generation call, and this post is explicit about where that line falls.

Why Scrapeless MCP

The Scrapeless MCP Server exposes web-scraping and browser tools an agent can call directly, so you do not build or host the scraping layer yourself. One connection serves 21 tools: scrape_markdown and scrape_html for content, google_search and google_trends for search data, scrape_screenshot for captures, and a full browser_* set that drives a cloud browser. AutoGen turns each of them into a native tool through mcp_server_tools, with no adapter code to write.

The browser_* tools drive the Scrapeless cloud browser, so an agent can navigate an interactive page and read what renders, all on Scrapeless infrastructure. For the same server wired into a different framework, the LangChain + Scrapeless MCP post covers the LangChain side.

Prerequisites

• Python 3.10 or later.
• A Scrapeless API key from the dashboard, exported as SCRAPELESS_API_KEY.
• A model-provider key (such as OPENAI_API_KEY) only for the agent run. Loading and calling the tools does not need one.

Install

Install AutoGen with the MCP extra and the agent package.

pip install "autogen-ext[mcp]" autogen-agentchat

Set your Scrapeless key in the shell. Use the real key at run time and keep the placeholder out of your source.

export SCRAPELESS_API_KEY="sk_your_key_here"

Load the Tools

StreamableHttpServerParams names the endpoint and carries the API key in the x-api-token header. mcp_server_tools runs the handshake and returns the server's tools as AutoGen tools.

import asyncio
import os

from autogen_ext.tools.mcp import StreamableHttpServerParams, mcp_server_tools


async def main() -> None:
    params = StreamableHttpServerParams(
        url="https://api.scrapeless.com/mcp",
        headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    )
    tools = await mcp_server_tools(params)
    names = sorted(tool.name for tool in tools)
    print("tool count:", len(names))
    print("tools:", ", ".join(names))


asyncio.run(main())

The live server returns 21 tools, loaded with only the Scrapeless key set.

tool count: 21
tools: browser_click, browser_close, browser_create, browser_get_html, browser_get_text, browser_go_back, browser_go_forward, browser_goto, browser_press_key, browser_screenshot, browser_scroll, browser_scroll_to, browser_snapshot, browser_type, browser_wait, browser_wait_for, google_search, google_trends, scrape_html, scrape_markdown, scrape_screenshot

The transport and message layer follow the Model Context Protocol specification, which rides on the JSON-RPC 2.0 specification. For a local server, AutoGen also ships StdioServerParams; the Scrapeless server is a hosted HTTP endpoint, so this guide uses the streamable-HTTP params.

Call a Tool

Each loaded tool is a StreamableHttpMcpToolAdapter, and you can call one directly before handing it to an agent. run_json takes the arguments and a cancellation token and returns the tool's result.

import asyncio
import os

from autogen_core import CancellationToken
from autogen_ext.tools.mcp import StreamableHttpServerParams, mcp_server_tools


async def main() -> None:
    params = StreamableHttpServerParams(
        url="https://api.scrapeless.com/mcp",
        headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    )
    tools = await mcp_server_tools(params)
    scrape_markdown = next(tool for tool in tools if tool.name == "scrape_markdown")
    result = await scrape_markdown.run_json({"url": "https://quotes.toscrape.com/"}, CancellationToken())
    text = result if isinstance(result, str) else str(result)
    print("markdown chars:", len(text))
    print("contains a quote:", "The world as we have created it" in text)


asyncio.run(main())

The call returns the page as Markdown, and the content check confirms a real quote is present.

markdown chars: 4424
contains a quote: True

This is the shape the agent gets back: page content it can reason over. The AutoGen MCP tools reference documents the params and adapter in full.

Attach the Tools to an Agent

Pass the tools to an AssistantAgent alongside a model client, and the agent calls them when the task needs them. This is the step that needs a model-provider key.

Note: AssistantAgent.run needs a model-provider key such as OPENAI_API_KEY, which is not set here. Loading the 21 tools and the direct scrape_markdown call above run without it. This block is shown with its exact shape; only the model round-trip is a prerequisite gap.

from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient


async def run_agent(tools) -> None:
    model_client = OpenAIChatCompletionClient(model="gpt-4o")
    agent = AssistantAgent("web_agent", model_client=model_client, tools=tools)
    result = await agent.run(
        task="Use scrape_markdown to fetch https://quotes.toscrape.com/ and list the first three quotes with authors."
    )
    print(result.messages[-1].content)

At run time the model reads the task, calls scrape_markdown with the URL, receives the Markdown the direct call already demonstrated, and writes the answer. The tools are the same objects whether the model calls them or you do.

Conclusion

AutoGen plus the Scrapeless MCP Server is a short path from a bare agent to one that reads the live web. mcp_server_tools loads the 21 tools, run_json proves one works, and one tools argument on the AssistantAgent attaches them all. Only the agent's generation step needs a model key, so you can load and test the entire tool surface first. Start from the scripts above, scope the tools to what the agent needs, and let the model drive.

Create a free Scrapeless account to get an API key, and check Scrapeless pricing when you plan a recurring agent.

FAQ

Q: Does AutoGen need a model client to load MCP tools?

No. mcp_server_tools runs the handshake and returns the tools with only the Scrapeless API key set, and each tool can be called directly with run_json. A model client is required only when you attach the tools to an AssistantAgent and call run, because that is when the model decides which tools to call.

Q: How do I call an MCP tool without an agent?

Each tool from mcp_server_tools is a StreamableHttpMcpToolAdapter with a run_json method. Pass a dictionary of arguments and a CancellationToken, and it returns the tool result. This is the quickest way to confirm the connection and inspect a tool's output before wiring it into an agent.

Q: How do I connect to a stdio MCP server instead of HTTP?

Swap the params. Use StdioServerParams with the server command instead of StreamableHttpServerParams with a URL, then pass it to the same mcp_server_tools call. The Scrapeless MCP Server is a hosted HTTP endpoint, so this guide uses the streamable-HTTP params.

Q: How do I give an agent only some of the tools?

mcp_server_tools returns a list, so filter it before passing it to the AssistantAgent. Handing an agent only scrape_markdown and google_search is safer than the full 21-tool set when the task needs just content and search.

Q: Which tools does the Scrapeless server provide?

Twenty-one: scrape_markdown, scrape_html, scrape_screenshot, google_search, google_trends, and a 16-tool browser_* set for navigation, clicking, typing, scrolling, and waiting on a cloud browser. Together they cover content extraction, search, and full browser interaction.

Q: Is scraping through the tools bound by the target's rules?

Yes. The tools fetch public pages, and you remain responsible for honoring each target's terms and its Robots Exclusion Protocol directives. Keep the volume bounded and the data public, and scope the agent to the tools the task actually needs.