CrewAI + Scrapeless: Give Your Agent Crew Live Web Data

A CrewAI crew is only as useful as the tools its agents can reach. Give a research agent nothing but a language model and it will confidently describe a page it never opened.

Connecting that crew to the Scrapeless MCP Server fixes the input side: agents get browser control, page scraping, Google Search, and Google Trends as ordinary CrewAI tools, while rendering, proxy routing, and anti-detection stay on the server. This guide runs the connection end to end and shows the tool list, the argument schemas, and the markdown a real call returns.

What This Setup Gives Your Crew

Your agents get 21 callable tools from one connection. The Scrapeless MCP Server exposes them over Streamable HTTP, and crewai-tools turns each one into a standard CrewAI BaseTool that any agent can hold.

The tools fall into three groups:

• Page retrieval — scrape_markdown, scrape_html, and scrape_screenshot fetch a URL and return it in the shape you asked for.
• Live browser control — sixteen browser_* tools that create a session and then click, type, scroll, navigate, wait, and snapshot inside it.
• Search surfaces — google_search and google_trends.

Because the work happens server-side, the crew process stays small. There is no local browser to install, no proxy pool to manage, and no driver version to keep aligned with a Chrome release.

Why the Scrapeless MCP Server

The Model Context Protocol specification defines how a client discovers and invokes tools on a server, which is what makes one connection worth more than a hand-written wrapper: the tool list, the argument schemas, and the result envelope all arrive from the server rather than being hard-coded in your project. Calls travel as JSON-RPC 2.0 messages, so the request and response format is a published standard rather than a vendor convention.

Scrapeless publishes a hosted endpoint, so there is no server to run. The transport is Streamable HTTP, the protocol's HTTP mechanism, and authentication is a single header. Everything a CrewAI agent needs is one dictionary. The same key also backs the Scrapeless Scraping Browser, which is the cloud browser the browser_* tools drive, and the parameter reference for each tool lives in the Scrapeless documentation.

Prerequisites

• Python 3.10 or later. Both crewai and crewai-tools currently declare >=3.10,<3.14.
• A Scrapeless API key from the dashboard.
• A model provider key for whichever LLM your crew runs on. CrewAI defaults to OpenAI and reads OPENAI_API_KEY.

Note: The examples below were executed with a Scrapeless key but without a model-provider key. The MCP connection, tool discovery, argument schemas, tool invocation, and agent attachment all ran live. The final crew.kickoff() call is a prerequisite gap — it needs a model key, and the article marks that step rather than showing invented output.

Install

pip install "crewai==1.15.4" "crewai-tools[mcp]==1.15.4"

The [mcp] extra pulls in the mcp client library and mcpadapt, which is the layer that converts MCP tool definitions into framework-native tools.

If your environment already carries an OpenTelemetry stack, install those two packages together as shown rather than one at a time. crewai pins opentelemetry-sdk~=1.42, and a partially upgraded exporter set causes an import error before any of your code runs.

Set the key in your shell:

export SCRAPELESS_API_KEY="your_api_key_here"

Connect Over Streamable HTTP

MCPServerAdapter takes one dictionary describing the server. The headers entry carries the Scrapeless API key:

import os
from crewai_tools import MCPServerAdapter

server_params = {
    "url": "https://api.scrapeless.com/mcp",
    "transport": "streamable-http",
    "headers": {"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
}

with MCPServerAdapter(server_params) as tools:
    names = sorted(t.name for t in tools)
    print(f"tool count: {len(names)}")
    for n in names:
        print("  -", n)

Running it lists what the server actually offers:

tool count: 21
  - browser_click
  - browser_close
  - browser_create
  - browser_get_html
  - browser_get_text
  - browser_go_back
  - browser_go_forward
  - browser_goto
  - browser_press_key
  - browser_screenshot
  - browser_scroll
  - browser_scroll_to
  - browser_snapshot
  - browser_type
  - browser_wait
  - browser_wait_for
  - google_search
  - google_trends
  - scrape_html
  - scrape_markdown
  - scrape_screenshot

Two details worth noting. The names are flat — there is no server prefix or dotted namespace, so scrape_markdown is the literal string an agent will call. And the context manager matters: it opens the session on entry and closes it on exit, which is why the adapter is written as a with block rather than a bare constructor.

Give an Agent Only the Tools It Needs

Handing all 21 tools to every agent makes the model's job harder, not easier. MCPServerAdapter accepts tool names after the server dictionary and returns only those:

import os
from crewai_tools import MCPServerAdapter

server_params = {
    "url": "https://api.scrapeless.com/mcp",
    "transport": "streamable-http",
    "headers": {"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
}

with MCPServerAdapter(server_params, "scrape_markdown", "google_search") as tools:
    print("filtered tools:", [t.name for t in tools])
    for t in tools:
        schema = getattr(t, "args_schema", None)
        fields = list(schema.model_fields) if schema else "n/a"
        print(f"  {t.name} args: {fields}")

filtered tools: ['scrape_markdown', 'google_search']
  scrape_markdown args: ['url']
  google_search args: ['q', 'hl', 'gl']

The argument schemas come from the server, so they are the real contract: scrape_markdown takes a single url, and google_search takes a query plus language and country codes. A research agent that only needs to read pages and run searches gets exactly two tools and no browser-session surface to misuse.

Ready to wire this into your own crew? Create a free Scrapeless account and connect with the key from your dashboard.

Attach the Tools to a Crew

Tools go straight into the Agent constructor, and the agent goes into a Crew with its task:

import os
from crewai import Agent, Task, Crew
from crewai_tools import MCPServerAdapter

server_params = {
    "url": "https://api.scrapeless.com/mcp",
    "transport": "streamable-http",
    "headers": {"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
}

with MCPServerAdapter(server_params, "scrape_markdown", "google_search") as tools:
    analyst = Agent(
        role="Web Research Analyst",
        goal="Turn public pages into clean markdown for downstream analysis.",
        backstory="Works with public web sources and hands back structured notes.",
        tools=tools,
        verbose=False,
    )
    print("agent tools:", [t.name for t in analyst.tools])

    task = Task(
        description="Fetch https://quotes.toscrape.com/js/ and summarise the authors present.",
        expected_output="A list of author names found on the page.",
        agent=analyst,
    )
    crew = Crew(agents=[analyst], tasks=[task], verbose=False)
    print("crew agents:", len(crew.agents), "| crew tasks:", len(crew.tasks))

agent tools: ['scrape_markdown', 'google_search']
crew agents: 1 | crew tasks: 1

The agent is holding both server-provided tools and the crew is assembled. Everything up to this point runs on the Scrapeless key alone.

Note: crew.kickoff() is the one step below that needs a model-provider key. Without OPENAI_API_KEY set, CrewAI raises ValueError: OPENAI_API_KEY is required before the first model call, so the run is shown as the line you add rather than as captured output.

    result = crew.kickoff()
    print(result)

What a Tool Call Returns

Calling a tool directly is the quickest way to see the returned shape without spending model tokens. scrape_markdown takes the URL and hands back markdown:

import os
from crewai_tools import MCPServerAdapter

server_params = {
    "url": "https://api.scrapeless.com/mcp",
    "transport": "streamable-http",
    "headers": {"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
}

with MCPServerAdapter(server_params, "scrape_markdown") as tools:
    tool = list(tools)[0]
    md = tool.run(url="https://quotes.toscrape.com/js/")
    text = md if isinstance(md, str) else str(md)
    print("markdown chars:", len(text))
    print("contains Einstein:", "Einstein" in text)
    print("first 180:", text[:180].replace("\n", " "))

markdown chars: 1580
contains Einstein: True
first 180: Response:  "# [Quotes to Scrape](https://quotes.toscrape.com/)\n\n[Login](https://quotes.toscrape.com/login)\n\n“The world as we have created it is a process of our thinking. It ca

The target page builds its quote list in the browser rather than sending it in the initial HTML, and the quotes are present in the markdown anyway — the server rendered the page before converting it. That is the practical difference between an MCP tool backed by real infrastructure and a plain HTTP fetch: the agent asks for a page and gets the page a user would see.

Markdown is also the format a language model handles most cheaply. Headings, links, and paragraph structure survive, while scripts, styling, and layout markup do not, so the agent spends its context on content.

Conclusion

Connecting CrewAI to the Scrapeless MCP Server takes one dictionary and a context manager. The server supplies 21 tools with their own argument schemas, crewai-tools converts them into native CrewAI tools, and naming specific tools in the adapter keeps each agent's surface small enough for a model to use well.

The part worth carrying over into your own project is the tool filter. A crew where the research agent holds scrape_markdown and google_search, and a separate browsing agent holds the browser_* set, gives each model a short menu and a clear job.

Start on the Scrapeless free plan to get a key, review the Scrapeless pricing when you size a workload, and read the Scrapeless MCP Server overview for the full tool reference.

FAQ

Q: What is the Scrapeless MCP Server endpoint for CrewAI?

The hosted endpoint is https://api.scrapeless.com/mcp, reached over the streamable-http transport with your key in the x-api-token header. CrewAI needs no local server process, because the tools are served remotely.

Q: How many tools does the Scrapeless MCP Server expose?

A live connection returns 21 tools: sixteen browser_* session-control tools, three page-retrieval tools (scrape_markdown, scrape_html, scrape_screenshot), and two search tools (google_search, google_trends). Check the list at runtime rather than assuming, since a server can add tools between releases.

Q: Can I limit which MCP tools an agent receives?

Yes. Pass tool names to MCPServerAdapter after the server dictionary — MCPServerAdapter(server_params, "scrape_markdown", "google_search") returns only those two. This keeps the model's tool menu short, which usually improves selection accuracy.

Q: Does CrewAI need an LLM key to connect to an MCP server?

No. The MCP handshake, tool discovery, and direct tool calls all work with only the Scrapeless key. A model-provider key is needed the moment you call crew.kickoff(), because that is when an agent asks a model which tool to use.

Q: Why use a context manager with MCPServerAdapter?

The with block opens the MCP session on entry and closes it on exit. Constructing the adapter without one leaves the connection open, and the tools are only valid while the session lives — accessing them after the session closes raises an error.

Q: Does scrape_markdown handle pages that render in the browser?

Yes. A page that writes its content in via JavaScript still comes back with that content in the markdown, because rendering happens server-side before conversion. A plain HTTP fetch of the same URL returns the pre-render markup instead.

Q: What should I check before pointing a crew at a live site?

Review the site's terms and its /robots.txt directives, which follow the Robots Exclusion Protocol standard. Keep collection to public pages, and give the crew a bounded task list rather than an open-ended crawl instruction — an agent loop can otherwise issue far more requests than you intended.