CrewAI + Scrapeless: Run a Multi-Agent Scraping Crew With a Local LLM

TL;DR:

• This is a real multi-agent workflow. Three CrewAI agents fetch a live page, extract structured records, and validate the result through a sequential task handoff.
• Only the fetcher gets web access. The Web Page Fetcher receives Scrapeless MCP’s scrape_markdown tool; the extractor and validator work only from previous task outputs.
• The LLM runs locally. All three agents use qwen2.5:0.5b through Ollama, so the workflow requires no cloud LLM API key.
• Successful execution does not prove correct extraction. crew.kickoff() completed, but the captured validator output contradicted itself and had to be rejected.
• Production validation must be deterministic. Parse the model output in Python, validate required fields, and compare extracted values with the original MCP response.
• Free to start. New Scrapeless accounts include free Scraping Browser runtime — sign up at app.scrapeless.com.

Introduction: A Completed Crew Is Not Yet a Reliable Pipeline

A CrewAI Agent connected to a Scrapeless MCP tool can already fetch a live page. A CrewAI crew is a different claim: two or more agents with separate responsibilities must hand work to one another and finish with crew.kickoff() returning a result.

This guide builds that second workflow.

You will create a three-agent crew that:

1. Fetches a live page through the Scrapeless MCP Server.
2. Extracts structured quote records from the returned Markdown.
3. Validates those records before they leave the workflow.

All three agents run on a local Ollama model. No OpenAI, Anthropic, or other cloud LLM key is required. The only external credential is the Scrapeless API key used by the MCP tool.

The integration does complete successfully. However, the captured final answer is internally inconsistent. That distinction is the most important lesson in this tutorial: a completed agent workflow is evidence of execution, not evidence that the resulting data is trustworthy.

What You Can Do With This Crew

The workflow assigns one responsibility to each agent:

• Web Page Fetcher: Calls the Scrapeless MCP scrape_markdown tool against a live URL and returns the page content.
• Data Extraction Specialist: Reads the fetched Markdown and converts it into a JSON array.
• QA Validator: Checks the extracted records for missing fields and reports the result.

CrewAI passes task outputs through explicit task context. The extractor receives the fetch task’s output, and the validator receives the extraction task’s output.

Nothing needs to be manually copied between the three agents.

This differs from wiring one local model directly into a fetch-and-extract Python function. The Ollama web scraping guide covers that simpler pattern.

Here, CrewAI owns the orchestration:

• Three agents
• Three scoped prompts
• Three separate token limits
• One framework-managed sequential handoff

Why Use a Crew Instead of One Agent?

A single agent with a scraping tool must decide what to fetch, how to interpret the page, how to format the result, and whether that result is acceptable.

Splitting those responsibilities into separate roles gives you clearer control over the workflow.

Each agent receives:

• One narrow goal
• One focused task
• Its own output limit
• Access only to the tools it needs

This can be especially helpful with smaller local models. Instead of asking one model turn to fetch, extract, format, and validate simultaneously, each turn handles a more limited job.

The separation also gives you clearer inspection points. In a production pipeline, you can save or validate the output after each task and identify whether a failure occurred during retrieval, extraction, or validation.

Why the Scrapeless MCP Server?

The Model Context Protocol specification defines a standard way for an AI client to discover and invoke tools exposed by a server.

The Scrapeless MCP Server exposes web data and browser capabilities through that tool interface. CrewAI does not need to implement page rendering, proxy routing, or browser infrastructure directly. It only needs to connect to the server and attach the required tool to an agent.

The Scrapeless MCP Server overview explains the broader tool family, including page retrieval, search, and browser-control tools.

For this workflow, the crew needs only one tool:

scrape_markdown

It retrieves the target page and returns content in a format the downstream extraction agent can read.

You can check the Scrapeless developer documentation for the latest server connection details and tool arguments.

Prerequisites

Before running the crew, you need:

• Python 3.10 or later
• CrewAI and CrewAI Tools
• A Scrapeless API key
• Ollama installed and running locally
• The qwen2.5:0.5b model downloaded in Ollama

You do not need OPENAI_API_KEY, ANTHROPIC_API_KEY, or another hosted LLM credential. Every CrewAI LLM object in this example points to the local Ollama server.

Step 1 — Install CrewAI and MCP Support

The verified example uses pinned CrewAI packages:

pip install "crewai==1.15.4" "crewai-tools[mcp]==1.15.4"

The [mcp] extra installs the MCP client dependencies needed by MCPServerAdapter. The adapter converts MCP tool definitions into tools that CrewAI agents can call.

Step 2 — Configure the Local Ollama Model

Pull the model used in this example:

ollama pull qwen2.5:0.5b

Confirm that Ollama can see it:

ollama list

Ollama normally exposes its local API at:

http://localhost:11434

If Ollama is installed but the crew cannot connect, confirm that the Ollama service is running before launching the Python script.

Next, configure the Scrapeless API key in your shell:

export SCRAPELESS_API_KEY="your_api_key_here"

Reading the key from an environment variable keeps it out of the Python source file.

Step 3 — Point CrewAI at Ollama

Create a separate LLM configuration for each agent:

from crewai import LLM

fetch_llm = LLM(
    model="ollama/qwen2.5:0.5b",
    base_url="http://localhost:11434",
    max_tokens=180,
    temperature=0,
)

extract_llm = LLM(
    model="ollama/qwen2.5:0.5b",
    base_url="http://localhost:11434",
    max_tokens=100,
    temperature=0,
)

validate_llm = LLM(
    model="ollama/qwen2.5:0.5b",
    base_url="http://localhost:11434",
    max_tokens=70,
    temperature=0,
)

All three configurations use the same model, but their output limits reflect their responsibilities:

• The fetcher has more room to return page content.
• The extractor needs enough tokens for a short JSON array.
• The validator only needs to produce a compact report.

Setting temperature=0 reduces output variation. It does not guarantee deterministic or correct results, especially with a small local model.

On CPU-only local inference, max_tokens is also an important runtime control. Lower limits prevent an agent from generating unnecessarily long responses, although model size, hardware, context length, and the number of agent turns also affect execution time.

Step 4 — Connect CrewAI to the Scrapeless MCP Server

Import MCPServerAdapter and define the remote MCP connection:

import os
from crewai_tools import MCPServerAdapter

server_params = {
    "url": "https://api.scrapeless.com/mcp",
    "transport": "streamable-http",
    "headers": {
        "x-api-token": os.environ["SCRAPELESS_API_KEY"]
    },
}

The configuration contains three important values:

• url points to the Scrapeless MCP endpoint.
• transport tells the client to use Streamable HTTP.
• x-api-token authenticates the request with your Scrapeless key.

The key is accessed with:

os.environ["SCRAPELESS_API_KEY"]

Python will stop immediately if the environment variable is missing, which is preferable to silently launching a crew without valid tool credentials.

Step 5 — Verify the Available MCP Tool

Before building the complete crew, verify that the adapter can discover the requested tool:

with MCPServerAdapter(server_params, "scrape_markdown") as tools:
    print([tool.name for tool in tools])

The verification run returned:

['scrape_markdown']

This confirms that:

• The MCP client reached the server.
• The server accepted the authentication header.
• The requested tool was available to CrewAI.

It does not prove that every future page request will contain the expected data, or that a downstream model will interpret the response correctly.

The MCP Tool Surface Used Here

MCPServerAdapter can expose server tools to CrewAI agents. Passing "scrape_markdown" limits this workflow to the single tool it needs:

with MCPServerAdapter(server_params, "scrape_markdown") as tools:
    ...

This narrower tool surface is useful for agent reliability. The fetcher does not need to select from unrelated browser or search tools, and the extraction and validation agents receive no tools at all.

The permission boundary is straightforward:

Only the fetcher can make a live web request.

How You Actually Use It: Build and Run the Crew

Define the Three Agent Roles

Create one Agent for each stage:

from crewai import Agent

fetcher = Agent(
    role="Web Page Fetcher",
    goal=(
        "Fetch the exact URL given using the scrape_markdown "
        "tool and hand back its raw output."
    ),
    backstory=(
        "Retrieves public web pages for teammates who cannot "
        "browse the web themselves."
    ),
    tools=tools,
    llm=fetch_llm,
    max_iter=2,
)

extractor = Agent(
    role="Data Extraction Specialist",
    goal=(
        "Turn fetched page markdown into a clean structured "
        "record of every quote and its author."
    ),
    backstory=(
        "Reads raw scraped markdown and pulls out exactly the "
        "fields a database needs."
    ),
    llm=extract_llm,
    max_iter=2,
)

validator = Agent(
    role="QA Validator",
    goal="Check extracted quote records for completeness before they ship.",
    backstory=(
        "Rejects incomplete or malformed records and reports "
        "exactly what it checked."
    ),
    llm=validate_llm,
    max_iter=2,
)

Only fetcher receives tools=tools.

The extractor and validator must work from task context. They cannot independently browse the target page or make another MCP request.

Why Set max_iter=2?

max_iter limits the number of reasoning and action cycles an agent can perform during a task.

A value of 2 gives the fetcher enough room to request a tool call and then produce a final answer. It also prevents a small model from continuing through an excessive number of internal loops.

The limit is a control mechanism, not a correctness guarantee. If an agent produces malformed JSON or accepts empty fields, reaching the iteration limit successfully does not make that output valid.

Attach the MCP Tool Only to the Fetcher

The fetcher’s role is narrowly defined:

1. Receive the target URL.
2. Call scrape_markdown.
3. Return the tool output without additional commentary.

The extractor never receives the live URL as a browsing instruction. It reads the fetcher’s returned content.

The validator never receives the MCP tool. It reads only the extractor’s output.

That separation makes the data flow easier to understand and audit.

Wire Sequential Task Context

CrewAI tasks can reference previous tasks through the context argument:

extract_task = Task(
    ...,
    context=[fetch_task],
)

validate_task = Task(
    ...,
    context=[extract_task],
)

The handoff is therefore:

fetch_task → extract_task → validate_task

The extractor receives the fetch task’s final answer. The validator receives the extraction task’s final answer.

No manual string-passing is required.

Run the Crew With crew.kickoff()

The following is the complete script:

import os

from crewai import Agent, Crew, LLM, Process, Task
from crewai_tools import MCPServerAdapter


TARGET_URL = "https://quotes.toscrape.com/tag/obvious/"

server_params = {
    "url": "https://api.scrapeless.com/mcp",
    "transport": "streamable-http",
    "headers": {
        "x-api-token": os.environ["SCRAPELESS_API_KEY"]
    },
}

fetch_llm = LLM(
    model="ollama/qwen2.5:0.5b",
    base_url="http://localhost:11434",
    max_tokens=180,
    temperature=0,
)

extract_llm = LLM(
    model="ollama/qwen2.5:0.5b",
    base_url="http://localhost:11434",
    max_tokens=100,
    temperature=0,
)

validate_llm = LLM(
    model="ollama/qwen2.5:0.5b",
    base_url="http://localhost:11434",
    max_tokens=70,
    temperature=0,
)

with MCPServerAdapter(server_params, "scrape_markdown") as tools:
    fetcher = Agent(
        role="Web Page Fetcher",
        goal=(
            f"Fetch the exact URL {TARGET_URL} using the "
            "scrape_markdown tool and hand back its raw output."
        ),
        backstory=(
            "Retrieves public web pages for teammates who "
            "cannot browse the web themselves."
        ),
        tools=tools,
        llm=fetch_llm,
        max_iter=2,
    )

    extractor = Agent(
        role="Data Extraction Specialist",
        goal=(
            "Turn fetched page markdown into a clean structured "
            "record of every quote and its author."
        ),
        backstory=(
            "Reads raw scraped markdown and pulls out exactly "
            "the fields a database needs."
        ),
        llm=extract_llm,
        max_iter=2,
    )

    validator = Agent(
        role="QA Validator",
        goal=(
            "Check extracted quote records for completeness "
            "before they ship."
        ),
        backstory=(
            "Rejects incomplete or malformed records and "
            "reports exactly what it checked."
        ),
        llm=validate_llm,
        max_iter=2,
    )

    fetch_task = Task(
        description=(
            f"Call the scrape_markdown tool with url='{TARGET_URL}'. "
            "Return the tool output exactly as your final answer, "
            "with no commentary before or after it."
        ),
        expected_output=(
            "The raw markdown returned by the scrape_markdown tool."
        ),
        agent=fetcher,
    )

    extract_task = Task(
        description=(
            "You are given the raw markdown of a quotes page as "
            "context. Find every quote on the page. For each one, "
            'output one JSON object with keys "text" (the quote) '
            'and "author" (the person named after "by"). Return '
            "ONLY a JSON array of these objects, nothing else."
        ),
        expected_output=(
            'A JSON array like [{"text": "...", "author": "..."}] '
            "with one entry per quote found."
        ),
        agent=extractor,
        context=[fetch_task],
    )

    validate_task = Task(
        description=(
            "You are given a JSON array of quote records as context. "
            "Check that the array has at least one entry and that "
            "every entry has a non-empty text and author field. "
            "Report the total record count, the author list, and "
            "a completeness statement."
        ),
        expected_output=(
            "A short report: record count, author list, "
            "completeness statement."
        ),
        agent=validator,
        context=[extract_task],
    )

    crew = Crew(
        agents=[fetcher, extractor, validator],
        tasks=[fetch_task, extract_task, validate_task],
        process=Process.sequential,
    )

    result = crew.kickoff()

print(result)

The script calls crew.kickoff() inside the MCP adapter context. This keeps the remote tool connection available while the fetcher performs its task.

After all three tasks finish, the script prints the final CrewOutput.

What You Get Back

The verification run produced evidence at three different layers:

The final validator output was:

0  ["", "", ""] Complete. The JSON array contains one entry and all fields are non-empty. Therefore, it meets the criteria for completeness.

This output is internally inconsistent.

It reports:

• A count of 0
• Three empty strings
• One supposedly complete entry
• A statement that all fields are non-empty

Those statements cannot all be true at the same time.

The process completed, but the data did not pass a meaningful quality check.

Why the Final Output Failed Validation

What the MCP Handshake Proved

The MCP handshake returned:

['scrape_markdown']

This proves that CrewAI connected to the Scrapeless MCP Server and discovered the requested tool.

The Web Page Fetcher therefore had a real remote tool available during execution.

What crew.kickoff() Proved

crew.kickoff() returned a CrewOutput without raising an exception.

This proves that:

• The crew was constructed successfully.
• The three tasks were accepted.
• Sequential orchestration reached the validator task.
• CrewAI returned a final result.

It does not prove that the fetched content survived every model handoff accurately.

What the Final Answer Did Not Prove

The target page contains one quote under the obvious tag:

“A day without sunshine is like, you know, night.”
— Steve Martin

The captured final answer did not preserve those values.

Because the script prints only the crew’s final output, it is not possible to assign the failure to one specific stage. The empty values might have appeared when:

• The fetcher summarized or altered the MCP response.
• The extractor converted the Markdown into JSON.
• The validator interpreted the extractor’s output.
• More than one stage degraded the data.

The available evidence supports only a narrower conclusion: the crew reached its final task, but the validator accepted a result that contradicted itself.

Small Local Models in Tool-Calling Crews: The Honest Result

The tested model contains approximately 494 million parameters. It was small enough to run locally, and it carried the CrewAI workflow through to completion.

The complete run exited with code 0 after 542 seconds on the tested host.

That timing is not a general CrewAI or Ollama benchmark. It reflects one machine, one model, one page, the selected prompts, and the number of agent turns in this particular workflow.

The integration result is still useful:

• CrewAI created three agents.
• The fetcher received a live Scrapeless MCP tool.
• Task context connected the three stages.
• crew.kickoff() completed.
• No cloud LLM key was used.

However, the run does not support trusting the resulting fields. The final output combined a zero count, empty values, and a success statement.

A sub-1B model can be useful for testing orchestration, but this run shows why it should not automatically be treated as a reliable structured-data extractor.

Add Deterministic Validation After the Crew

Natural-language validation is still model output. A validator agent can misunderstand malformed data, overlook empty values, or generate a conclusion that conflicts with its own report.

Production validation should therefore happen in ordinary code after the model workflow.

Parse the Extractor Output

Use json.loads to parse the extractor’s response.

Reject the result if:

• It is not valid JSON.
• The top-level value is not an array.
• The response includes prose around the JSON.
• The array is unexpectedly empty.

Validate the Required Fields

For every record, verify that:

• text exists.
• author exists.
• Both values are strings.
• Both values remain non-empty after trimming whitespace.
• Neither value is an obvious placeholder.

Do not let a natural-language statement such as “all fields are complete” override a failed programmatic check.

Reject Contradictory Counts

If the validator reports a count, compare it with the actual JSON array length.

A report claiming one complete entry while also reporting a count of zero should fail immediately.

Compare Extracted Values With the Source

Keep the original Markdown returned by scrape_markdown.

For copied fields such as quotes and author names, confirm that the extracted values appear in the source response. This helps detect hallucinated, truncated, or substituted content.

Preserve Intermediate Task Outputs

The example prints only the final crew output. For debugging and production monitoring, preserve the output from every task.

That gives you three separate artifacts:

1. Raw fetched Markdown
2. Extracted JSON
3. Validation report

With those outputs available, you can identify the exact stage where data was lost instead of inferring from the final answer.

Increase Model Capacity When Necessary

Stronger validation can reject bad results, but it cannot restore source text that a model failed to preserve.

If a small model repeatedly loses data during extraction or validation, use a larger local model or a more capable hosted model. Do not weaken the checks simply to make the pipeline appear successful.

Conclusion

A CrewAI crew running on a local Ollama model can connect to the Scrapeless MCP Server, call a live scraping tool, pass data through three agents, and complete crew.kickoff() without a cloud LLM key.

That is the integration result.

The captured final output was still wrong enough to reject: it reported a zero count and empty values while simultaneously claiming the data was complete.

Treat workflow completion and data correctness as separate conditions. Preserve intermediate outputs, validate structured records in Python, compare extracted values with the MCP response, and fail the pipeline whenever those checks disagree with the model’s conclusion.

Ready to Build a Tool-Calling Crew?

Create a free Scrapeless account to get your API key and connect CrewAI to a live web data tool.

Once you understand the number of pages and agent calls your workflow requires, review the Scrapeless pricing plans.

For implementation questions and community support:

• Join the Scrapeless Discord
• Join the Scrapeless Telegram community

FAQ

Q: Does a CrewAI crew need a cloud LLM key to run?

No. Setting model="ollama/qwen2.5:0.5b" and base_url="http://localhost:11434" points each agent at the local Ollama server. The crew does not need an OpenAI, Anthropic, or other hosted LLM key.

This workflow still needs a Scrapeless API key because the fetcher calls the remote Scrapeless MCP Server.

Q: How does CrewAI pass output from one agent to the next?

Each downstream Task receives a context list containing an earlier task.

For example:

extract_task = Task(
    ...,
    context=[fetch_task],
)

CrewAI includes the fetch task’s final answer in the extractor’s context. The same mechanism passes the extraction result to the validator.

Q: Why does only the fetcher receive the MCP tool?

The fetcher is the only agent responsible for retrieving live page content. The extractor and validator should operate on existing task outputs.

Limiting tool access reduces unnecessary choices and makes the workflow easier to audit.

Q: What does max_iter=2 control?

max_iter limits how many reasoning and action cycles an agent can perform during one task.

A value of 2 gives the fetcher room for a tool call and a final response while preventing an open-ended loop. It limits execution, but it does not guarantee that the agent’s final answer is correct.

Q: Did the small local model extract the correct quote?

No. The crew completed, but the final validator output began with:

0  ["", "", ""]

It then claimed that one complete entry existed and all fields were non-empty. Because those statements contradict one another, the result must be rejected.

Q: Why use deterministic validation if the crew already has a QA agent?

A QA agent is still an LLM. It can overlook missing fields or generate a conclusion that conflicts with the data it was asked to inspect.

Deterministic Python checks provide an independent pass/fail decision based on JSON syntax, array length, required fields, and source matching.

Q: How slow was the local Ollama run?

The complete verification run took 542 seconds on the tested host. That measurement applies only to the specific hardware, model, page, prompts, and agent configuration used in the test.

Local CPU inference can take minutes when several agent turns are involved.

Q: What should be logged in a production crew?

Preserve at least:

• The MCP tool response
• Every intermediate task output
• The parsed structured data
• Deterministic validation errors
• The final crew result
• Execution timing for each stage

This makes it possible to locate the stage that altered or dropped the source data.

Q: What should I check before scraping a live website?

Review the website’s terms and /robots.txt directives, which follow the Robots Exclusion Protocol.

Keep the target list bounded, use public pages, and avoid giving an autonomous agent an open-ended crawling instruction.