How to Build an AI Agent in Python

TL;DR:

• An AI agent is a loop: the model picks a tool, your code runs it, the result goes back to the model. The intelligence is the language model; the reach comes from the tools you give it. For research and monitoring, the two tools that matter are web search and page fetch.
• Give the agent real web tools, not a frozen training set. A web_search tool backed by Deep SerpApi returns live Google results, and a fetch_page tool backed by the Universal Scraping API returns rendered HTML — so the agent reasons over current data, not last year's.
• Both tools are one HTTP POST each. Search hits POST /api/v1/scraper/request with the scraper.google.search actor; fetch hits POST /api/v1/unlocker/request with js_render on. No browser to manage, no proxy pool to rotate.
• The model owns the control flow. You expose the tool signatures, the model decides what to call and when, and you execute and feed results back until it answers — that loop is the whole agent.
• Keep the tool layer deterministic and verified. The search and fetch calls return structured data you can test independently of the model, which is where reliability comes from.
• Free to start. New Scrapeless accounts include free Deep SerpApi runtime — sign up at app.scrapeless.com.

Introduction: an agent is only as good as its tools

A language model on its own can reason, but it can't see today's web — it answers from training data with a cutoff. An AI agent closes that gap by wrapping the model in a loop: the model asks to run a tool, your code runs it against live data, and the result goes back into the conversation. Repeat until the model has what it needs to answer.

The hard part is rarely the loop — it's the tools. A research agent needs to search the open web and read the pages it finds, and both of those break on real sites: search engines rate-limit, and target pages render with JavaScript or sit behind bot defenses. Hand-rolling that is most of the work.

This guide builds a research agent in Python where both tools are managed HTTP calls: web_search runs on the Scrapeless Deep SerpApi and fetch_page runs on the Universal Scraping API. The tool layer below is verified against the live API; the model reasoning loop is shown with the standard tool-calling pattern. Public data only.

Pipeline at a glance

The agent has four parts, and only the middle two touch the web:

1. Goal in — a question or research task from the user.
2. web_search(query) — live Google results via Deep SerpApi (title, link, snippet).
3. fetch_page(url) — rendered HTML for any result the model wants to read, via the Universal Scraping API.
4. Model loop — the language model decides which tool to call, reads the result, and either calls another tool or writes the final answer.

Steps 2 and 3 are deterministic HTTP calls you can verify on their own. Step 4 is the model; steps 1–3 are what make its answers current.

Prerequisites

• Python 3.10 or newer
• pip install requests (plus your model provider's SDK)
• A Scrapeless account and API key — sign up at app.scrapeless.com
• A model provider API key for the reasoning loop (Step 3)

export SCRAPELESS_API_KEY="your_api_token_here"

Step 1 — The web search tool

The search tool posts a query to the Deep SerpApi Google Search actor and returns a compact list of organic results — title, link, and snippet — which is exactly what a model needs to decide what to read next:

import os
import requests

API = "https://api.scrapeless.com"
HEADERS = {
    "Content-Type": "application/json",
    "x-api-token": os.environ["SCRAPELESS_API_KEY"],
}

def web_search(query: str, count: int = 5):
    r = requests.post(
        f"{API}/api/v1/scraper/request",
        headers=HEADERS,
        json={
            "actor": "scraper.google.search",
            "input": {"q": query, "hl": "en", "gl": "us"},
        },
        timeout=90,
    )
    r.raise_for_status()
    results = r.json().get("organic_results", [])[:count]
    return [
        {"title": x.get("title"), "link": x.get("link"), "snippet": x.get("snippet")}
        for x in results
    ]


# Run it on its own to confirm the shape:
hits = web_search("best web scraping tools 2026", 3)
print(len(hits), "results")
for h in hits:
    print("-", h["title"], "|", h["link"])

The actor returns organic_results with the standard search fields; the function trims it to the top count so you don't flood the model's context with the full SERP.

Get your API key on the free plan: app.scrapeless.com

Step 2 — The page fetch tool

Once the model picks a result to read, it needs the page content. The fetch tool posts the URL to the Universal Scraping API with js_render on, so client-rendered pages come back as finished HTML rather than an empty shell:

import os
import requests

API = "https://api.scrapeless.com"
HEADERS = {
    "Content-Type": "application/json",
    "x-api-token": os.environ["SCRAPELESS_API_KEY"],
}

def fetch_page(url: str):
    r = requests.post(
        f"{API}/api/v1/unlocker/request",
        headers=HEADERS,
        json={
            "actor": "unlocker.webunlocker",
            "input": {"url": url, "method": "GET", "js_render": True},
        },
        timeout=120,
    )
    r.raise_for_status()
    return r.json().get("data", "")


# Fetch a rendered page and confirm we get HTML back:
html = fetch_page("https://www.scrapeless.com/en")
print("fetched", len(html), "bytes")

Before handing the HTML to the model, strip it to text (with selectolax or BeautifulSoup) so you spend context on content, not markup. Keep the tool itself returning raw HTML — let the agent decide how much to read.

Step 3 — The model loop

With both tools verified, the agent is the loop that lets the model call them. Expose the two tool signatures to your model provider's tool-calling API, then run the standard cycle: send the conversation, and while the model returns a tool call, execute it and append the result; when it returns text, that's the answer.

Note: this step calls your model provider and requires that provider's API key. The tool functions above are fully runnable on their own; the loop below is the standard tool-calling pattern wired to them.

TOOLS = [
    {
        "name": "web_search",
        "description": "Search Google for a query; returns title, link, snippet.",
        "input_schema": {
            "type": "object",
            "properties": {"query": {"type": "string"}},
            "required": ["query"],
        },
    },
    {
        "name": "fetch_page",
        "description": "Fetch the rendered HTML of a URL.",
        "input_schema": {
            "type": "object",
            "properties": {"url": {"type": "string"}},
            "required": ["url"],
        },
    },
]

DISPATCH = {"web_search": web_search, "fetch_page": fetch_page}

def run_agent(client, goal: str):
    messages = [{"role": "user", "content": goal}]
    while True:
        reply = client.run(messages=messages, tools=TOOLS)  # provider tool-calling call
        if reply.tool_call:
            name, args = reply.tool_call.name, reply.tool_call.args
            result = DISPATCH[name](**args)
            messages.append({"role": "tool", "name": name, "content": result})
            continue
        return reply.text

The shape is provider-agnostic: every major tool-calling API gives you "the model wants to call tool X with args Y", you run DISPATCH[X](**Y), append the result, and loop. The agent's competence comes from the two tools returning real, current data — which is the part this guide verified.

What You Get Back

Each web_search result is a flat record the model can reason over directly:

[
  {
    "title": "Best Web Scraping Tools in 2026",
    "link": "https://dev.to/nitinfab/best-web-scraping-tools-in-2026-i-tested-30-tools-and-these-are-the-only-ones-worth-using-11l3",
    "snippet": "A hands-on comparison of scraping tools across rendering, proxies, and price ..."
  }
]
// Schema reflects exactly what web_search returns. Field values are illustrative samples.

A few honest observations:

• Trim search results before they hit the model. The top three to five are usually enough; the full SERP wastes context and money.
• Convert HTML to text in the agent, not the tool. Keep fetch_page returning raw HTML so it stays deterministic; do the cleanup where you control the token budget.
• Cap the loop. Give the agent a maximum tool-call count so a confused run can't spin — a hard cap, not a re-run.
• The tools are independently testable. Because search and fetch are plain HTTP, you can unit-test them without the model in the loop, which is where reliability lives.

Conclusion: tools make the agent

A research agent is a small loop around a capable model — the leverage is in the tools. Back web_search with Deep SerpApi and fetch_page with the Universal Scraping API, and the agent reasons over live Google results and rendered pages instead of stale training data, with no browser fleet or proxy rotation to maintain. Build the tools first, verify them on their own, then wrap the model loop around them. For an agent that drives a full browser instead of HTTP calls, see building a search agent on the Scraping Browser; the Deep SerpApi product page and docs cover every actor. Verify the tools, let the model drive, and cap the loop.

Ready to Build Your AI-Powered Data Pipeline?

Join our community to claim a free plan and connect with developers building agents and research pipelines: Discord · Telegram.

Sign up at app.scrapeless.com for free Deep SerpApi runtime and wire the search and fetch tools into the agent framework you already use. See pricing for scale.

FAQ

Q: Do I need a specific model provider to build this?
No. The loop is provider-agnostic — any tool-calling API works. Swap the client.run(...) call for your provider's; the web_search and fetch_page tools stay identical.

Q: Why use managed APIs for the tools instead of requests and a headless browser?
Because the open web fights raw scrapers: search engines rate-limit and target pages render client-side or gate behind challenges. Deep SerpApi and the Universal Scraping API handle rendering and access, so the tools return clean data instead of blocks.

Q: Is the agent's web access legal?
The agent collects publicly available data. How you store and use it is governed by each site's Terms of Service and local law — access only public data, respect the ToS, and consult counsel for your use case.

Q: How do I stop the agent from looping forever?
Cap the number of tool calls per run and stop when the limit is hit. That's a hard ceiling on control flow, not error handling.

Q: How big can fetched pages get?
Large — a single content page can return hundreds of kilobytes of HTML. Convert to text and truncate before passing it to the model so you control the context budget.

Q: Can the agent use more than two tools?
Yes. Add any tool that returns structured data — a Google Maps actor, a news search, a database query — to the TOOLS list and the DISPATCH map, and the model can call it the same way.