DSPy + Scrapeless: Give Your DSPy Program Live Web Tools via MCP

TL;DR:

• DSPy converts the Scrapeless MCP Server's tools into dspy.Tool objects with dspy.Tool.from_mcp_tool, giving a DSPy program all 21 web tools, from scrape_markdown to a full browser set.
• Each converted tool is bound to a live mcp.ClientSession, so the tools only work while that session is open; keep the whole flow inside one async with ClientSession(...) block.
• Converting the tools and calling one directly with acall run with no language-model configured; only the dspy.ReAct run needs an LM.
• dspy.Tool.from_mcp_tool(session, tool) takes the session and one MCP tool and returns a DSPy tool you can call or hand to a module.
• A direct scrape_markdown call returns the page as Markdown, ready to feed into a DSPy signature.
• Start on the Scrapeless free plan and give your DSPy program real web tools.

DSPy is built on a different idea from most agent frameworks: you declare what you want with a signature and let DSPy handle the prompting. Tools fit that model cleanly, but DSPy does not ship a way to reach the live web. The Model Context Protocol supplies it. Convert an MCP server's tools into DSPy tools and a dspy.ReAct module can call them the same way it calls any other tool.

This guide connects DSPy to the Scrapeless MCP Server, converts its 21 tools, calls one for real, and shows where a language-model key becomes necessary. The conversion and the direct call are verified against the live server; the module run is marked as the one prerequisite it needs.

Why Scrapeless MCP

The Scrapeless MCP Server exposes web-scraping and browser tools an agent can call directly, so the scraping layer is not something you build or host. One connection serves 21 tools: scrape_markdown and scrape_html for content, google_search and google_trends for search data, scrape_screenshot for captures, and a full browser_* set that drives a cloud browser. DSPy bridges to all of them through dspy.Tool.from_mcp_tool, which turns each MCP tool into a native DSPy tool.

The browser_* tools drive the Scrapeless cloud browser, so a program can navigate an interactive page and read what renders, all on Scrapeless infrastructure. For the protocol view of the same server, the MCP integration guide covers how MCP clients connect in general.

Prerequisites

• Python 3.10 or later.
• A Scrapeless API key from the dashboard, exported as SCRAPELESS_API_KEY.
• A language-model key (such as OPENAI_API_KEY) only for the dspy.ReAct run. Converting and calling the tools does not need one.

Install

Install DSPy and the MCP client library.

pip install dspy mcp

Set your Scrapeless key in the shell. Use the real key at run time and keep the placeholder out of your source.

export SCRAPELESS_API_KEY="sk_your_key_here"

Open a Session and Convert the Tools

DSPy's MCP bridge works on a live session. Open a streamable-HTTP connection, wrap it in an mcp.ClientSession, initialize it, list the server's tools, and convert each one with dspy.Tool.from_mcp_tool. The Scrapeless key rides in the x-api-token header.

import asyncio
import os

import dspy
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client


async def main() -> None:
    async with streamablehttp_client(
        "https://api.scrapeless.com/mcp", headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]}
    ) as (read, write, _):
        async with ClientSession(read, write) as session:
            await session.initialize()
            mcp_tools = (await session.list_tools()).tools
            tools = [dspy.Tool.from_mcp_tool(session, t) for t in mcp_tools]
            names = sorted(t.name for t in tools)
            print("dspy tools:", len(tools))
            print("tools:", ", ".join(names))


asyncio.run(main())

The live server yields 21 DSPy tools, converted with no language model configured.

dspy tools: 21
tools: browser_click, browser_close, browser_create, browser_get_html, browser_get_text, browser_go_back, browser_go_forward, browser_goto, browser_press_key, browser_screenshot, browser_scroll, browser_scroll_to, browser_snapshot, browser_type, browser_wait, browser_wait_for, google_search, google_trends, scrape_html, scrape_markdown, scrape_screenshot

Each converted tool holds a reference to session, which is why the conversion and everything that uses the tools stays inside the async with ClientSession(...) block. Close the session and the tools stop working. The transport and message layer follow the Model Context Protocol specification, which rides on the JSON-RPC 2.0 specification.

Call a Tool

A DSPy tool is callable on its own, so you can run one before building a module. acall invokes the tool with keyword arguments and returns its result.

import asyncio
import os

import dspy
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client


async def main() -> None:
    async with streamablehttp_client(
        "https://api.scrapeless.com/mcp", headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]}
    ) as (read, write, _):
        async with ClientSession(read, write) as session:
            await session.initialize()
            mcp_tools = (await session.list_tools()).tools
            tools = [dspy.Tool.from_mcp_tool(session, t) for t in mcp_tools]
            scrape_markdown = next(t for t in tools if t.name == "scrape_markdown")
            result = await scrape_markdown.acall(url="https://quotes.toscrape.com/")
            text = result if isinstance(result, str) else str(result)
            print("markdown chars:", len(text))
            print("contains a quote:", "The world as we have created it" in text)


asyncio.run(main())

The call returns the page as Markdown, and the content check confirms a real quote is present.

markdown chars: 4308
contains a quote: True

Calling a tool directly is the fastest way to confirm the connection and inspect what a tool returns, and it is the same object a module will call. The DSPy documentation covers tools, signatures, and modules in full.

Plug the Tools Into a Module

dspy.ReAct takes a signature and a list of tools and runs the reason-act loop. This is the step that needs a language model: configure one with dspy.configure, then let the module decide when to call scrape_markdown or any other tool. Because the tools are bound to the session, the module runs inside the same async with ClientSession(...) block that converted them.

Note: dspy.configure(lm=...) and the dspy.ReAct run need a language-model key such as OPENAI_API_KEY, which is not set here. Converting the 21 tools and the direct scrape_markdown call above run without it. This block is shown with its exact shape; only the model round-trip is a prerequisite gap.

# inside the `async with ClientSession(...)` block, after converting `tools`
dspy.configure(lm=dspy.LM("openai/gpt-4o"))
agent = dspy.ReAct("question -> answer", tools=tools)
result = await agent.acall(
    question="Fetch https://quotes.toscrape.com/ and list the first three quotes with their authors."
)
print(result.answer)

At run time the module reads the signature, calls scrape_markdown to get the page, reasons over the Markdown the direct call already demonstrated, and fills the answer field. The tools are the same objects whether the module calls them or you do.

Conclusion

DSPy plus the Scrapeless MCP Server keeps DSPy's declarative style while adding real web reach. dspy.Tool.from_mcp_tool converts the 21 tools, acall proves one works, and dspy.ReAct turns them into a running program. The one rule to remember is that the tools live on the session, so keep the flow inside one session block, and only the module run needs a model key. Start from the scripts above, scope the tools to what your signature needs, and let DSPy do the prompting.

Create a free Scrapeless account to get an API key, and check Scrapeless pricing when you plan a recurring program.

FAQ

Q: Does DSPy need a language model to load MCP tools?

No. Opening the session, listing the tools, converting them with dspy.Tool.from_mcp_tool, and calling one with acall all run with only the Scrapeless API key. A language-model key is required only for dspy.ReAct, when the module itself decides which tools to call.

Q: Why must the code stay inside one ClientSession block?

dspy.Tool.from_mcp_tool binds each tool to the mcp.ClientSession you pass it, so the tools issue their calls through that session. Once the async with ClientSession(...) block exits, the session closes and the tools can no longer run, which is why converting and using the tools belong in the same block.

Q: How is DSPy's MCP integration different from the adapter frameworks?

DSPy converts tools from a raw mcp.ClientSession with dspy.Tool.from_mcp_tool, rather than through a higher-level adapter that manages the connection for you. The trade-off is explicit control of the session lifetime in exchange for one fewer dependency, and it keeps the tools as ordinary dspy.Tool objects.

Q: How do I call a tool without building a module?

Every converted tool is callable with acall and keyword arguments, so await scrape_markdown.acall(url="...") returns the tool result directly. This is useful for confirming the connection and inspecting output before you wrap the tools in a dspy.ReAct module.

Q: How do I give a module only some of the tools?

from_mcp_tool runs per tool, so build the tools list from just the MCP tools you want, or filter the converted list before passing it to dspy.ReAct. Handing a module only scrape_markdown and google_search is safer than the full 21-tool set when the task needs just content and search.

Q: Is scraping through the tools bound by the target's rules?

Yes. The tools fetch public pages, and you remain responsible for honoring each target's terms and its Robots Exclusion Protocol directives. Keep the volume bounded and the data public, and scope the module to the tools the task actually needs.