Dify + Scrapeless: Give Your Agents Live Web Data With a Custom Tool

TL;DR:

• The Deep SerpApi plugin in the Dify Marketplace exposes exactly one tool with one parameter, query, so any request that needs a result vertical, a page offset, or a different site has to come from somewhere else.
• A custom tool is one OpenAPI file. Dify parses it into a single operation, scraperRequest, that reaches the whole Scrapeless scraper.* actor family through one endpoint.
• Dify pre-fills two authentication fields with values this API rejects: the header name defaults to Authorization and the header prefix defaults to Basic. Either default returns 401 with {"code":14404,"message":"invalid access token"}.
• Dify types the nested input object as a string parameter, so a Code node that emits JSON text is the reliable way to build it inside a Workflow.
• An Amazon product call returns about 2.2 MB, of which 1.9 MB is raw html. Select result in a Code node before the payload reaches a model.
• A free Scrapeless account covers every request in this guide.

A Dify agent with no web tool answers from its model weights and whatever you uploaded to its knowledge base. Ask it for today's top-ranking pages, a competitor's current price, or the plumbers operating in a specific city, and it will produce something fluent and stale.

Dify solves that with tools, and there are two ways to add one. This guide covers the second: a custom tool built from an OpenAPI file, which turns the Scrapeless Scraping API into a callable action in every agent and workflow in your workspace.

What a Custom Tool Adds That the Plugin Does Not

The official Deep SerpApi listing in the Dify Marketplace exposes one tool with a single required parameter, query, and one credential field for the API key. If a plain Google query is all your workflow needs, install it and stop reading — it is two clicks and it works, and the business news monitor built on Dify shows a full workflow assembled around it.

The Scrapeless HTTP endpoint behind it accepts considerably more than a query string. The same request shape selects the local pack instead of web results, offsets to the second page of those results, or switches to an Amazon listing entirely. None of that is reachable through a single query field.

A custom tool closes that gap. You paste an OpenAPI document, Dify reads the operations out of it, and the whole actor family becomes one attachable tool. There is nothing to install and nothing to deploy, and the same file works on Dify Cloud and on a self-hosted instance.

What the Scraping API Returns

One endpoint takes every request: POST https://api.scrapeless.com/api/v1/scraper/request. The body carries two fields — actor names the scraper, and input carries that scraper's parameters.

The response is parsed JSON rather than HTML. A scraper.google.search call puts organic_results at the top level next to metadata, pagination, and search_information. Adding tbm: lcl to the same actor swaps that for local_results.places, the block of businesses with ratings, phone numbers, and addresses. A scraper.amazon call nests the parsed product under result.

That single-shape design is what makes one OpenAPI operation enough. Details of every actor's parameters live in the Scraping API documentation.

Prerequisites

• A Dify workspace — Cloud, or self-hosted on 1.0.0 or later. The behavior described here was measured on a self-hosted 1.16.1 instance.
• A Scrapeless API key from the dashboard.
• Workspace permission to add tools. Dify restricts the custom-tool endpoints to workspace admins and owners.

Step 1: Import the OpenAPI Schema

In Dify, open Tools → Custom → Create Custom Tool and paste the document below. It is valid against the OpenAPI 3.0.3 specification, which is the version Dify's parser expects.

openapi: 3.0.3
info:
  title: Scrapeless Scraper API
  version: "1.0.0"
servers:
  - url: https://api.scrapeless.com
paths:
  /api/v1/scraper/request:
    post:
      operationId: scraperRequest
      summary: Run a scraper actor and return structured data
      security:
        - ApiTokenAuth: []
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [actor, input]
              properties:
                actor:
                  type: string
                  description: Which scraper to run.
                  enum: [scraper.google.search, scraper.amazon]
                  example: scraper.google.search
                input:
                  type: object
                  description: Actor parameters. Keys depend on the actor.
                  additionalProperties: true
            examples:
              googleSearch:
                summary: Google SERP
                value:
                  actor: scraper.google.search
                  input:
                    q: web scraping api
              googleLocalPack:
                summary: Google local pack
                value:
                  actor: scraper.google.search
                  input:
                    q: plumbers in Austin, TX
                    tbm: lcl
              amazonProduct:
                summary: Amazon product by URL
                value:
                  actor: scraper.amazon
                  input:
                    action: product
                    url: https://www.amazon.com/dp/B09B8V1LZ3
      responses:
        '200':
          description: Parsed result. Shape depends on the actor.
          content:
            application/json:
              schema:
                type: object
                additionalProperties: true
components:
  securitySchemes:
    ApiTokenAuth:
      type: apiKey
      in: header
      name: x-api-token

Dify parses that into exactly one tool. The name comes from operationId, so the tool is called scraperRequest, and it takes two parameters: actor and input. The three named examples appear in the request builder, which saves typing the Amazon URL by hand.

Step 2: Fill In All Four Authentication Fields

Choose API Key authentication and set every field. Two of the four arrive pre-filled with values this API rejects:

The prefix field is the one that catches people. Dify concatenates it onto the value, so leaving it on Basic sends the header x-api-token: Basic <your-key>. That is not what the Basic HTTP authentication scheme means — a real Basic credential is a base64-encoded user:password pair — and Scrapeless expects the bare key, so the request is rejected. Bearer fails identically. Only Custom passes the value through untouched.

Leaving the header name on Authorization fails the same way, for the same reason: the key never lands in the header the API reads.

Both mistakes produce one response, and you can reproduce either from a terminal before touching Dify:

# Correct: bare key in x-api-token
curl -s -o /dev/null -w 'bare key      -> %{http_code}\n' \
  -X POST https://api.scrapeless.com/api/v1/scraper/request \
  -H "Content-Type: application/json" \
  -H "x-api-token: $SCRAPELESS_API_KEY" \
  -d '{"actor":"scraper.google.search","input":{"q":"web scraping api"}}'

# What Dify sends with the default prefix
curl -s -w '\nBasic prefix  -> %{http_code}\n' \
  -X POST https://api.scrapeless.com/api/v1/scraper/request \
  -H "Content-Type: application/json" \
  -H "x-api-token: Basic $SCRAPELESS_API_KEY" \
  -d '{"actor":"scraper.google.search","input":{"q":"web scraping api"}}'

# What Dify sends with the default header name
curl -s -w '\nAuthorization -> %{http_code}\n' \
  -X POST https://api.scrapeless.com/api/v1/scraper/request \
  -H "Content-Type: application/json" \
  -H "Authorization: $SCRAPELESS_API_KEY" \
  -d '{"actor":"scraper.google.search","input":{"q":"web scraping api"}}'

bare key      -> 200
{"code":14404,"message":"invalid access token"}
Basic prefix  -> 401
{"code":14404,"message":"invalid access token"}
Authorization -> 401

A 401 here is the server telling you the credential it received is not one it accepts, which is exactly what the HTTP semantics specification reserves that status for. The body narrows it further: code 14404 is specifically an unusable token, not a malformed request.

Step 3: Run the Built-In Test

Dify's test panel calls the endpoint with the credentials you just entered. Fill the two parameters:

{
  "actor": "scraper.google.search",
  "input": "{\"q\": \"web scraping api\"}"
}

A working configuration returns roughly 15 KB of SERP JSON. A broken prefix returns a single error string that carries the upstream body verbatim: Request failed with status code 401 and {"code":14404,"message":"invalid access token"}.

Note the quoting in the test payload. Dify flattens nested request-body properties, so input is registered as a string parameter rather than an object — the parsed schema reports actor and input both as string, both required. A real JSON object works in the panel as well, because Dify normalizes either form into the object the API expects. That conversion matters: a request built by hand against the endpoint has to send an object, and a string there comes back as 400 {"message":"invalid input body"}.

Save the provider once the test returns data. scraperRequest then appears in the tool list for every app in the workspace.

Building this on a free plan? Create a Scrapeless account and the requests in this guide run on the free quota.

What Comes Back

The envelope depends on the actor, and each shape wants different handling downstream.

Web search. scraper.google.search with {"q": "web scraping api"} returned eight organic_results in a 15 KB response, alongside metadata, pagination, search_information, related_searches, and an inline_videos block. Each result carries title, link, snippet, source, position, and snippet_highlighted_words.

Local pack. Adding tbm: lcl replaces organic_results with local_results.places — 20 businesses per request. Setting start: 20 returns the next page; across two consecutive pages of one query, 37 of the 40 records were distinct, so a flow that stores both pages should key on something stable rather than assuming no repeats.

Local-pack fields need a cleaning pass before they reach a CRM or a spreadsheet:

• phone, type, and hours arrive padded with a leading space, and some hours strings use a narrow no-break space instead of a normal one.
• phone held a phone-shaped value in 15 of 20 records in one capture; the rest carried opening hours or a service label such as Online estimates.
• place_id, place_id_search, lsig, and thumbnail were empty in all 20 records.
• gps_coordinates is present but reads {"latitude": 0, "longitude": 0}, so it passes a truthiness check while carrying no location.

Amazon. scraper.amazon with action: product returned 2,226,755 bytes. The parsed product under result is 4,608 bytes across 63 fields; the remaining 1,960,588 bytes are the raw html of the listing. Handing that whole payload to a model is expensive and pointless.

Trim the Response Before It Reaches the Model

Put a Code node directly after the Tool node. It runs Python 3 or JavaScript, takes the tool output as an input variable, and returns a dict that later nodes read by key. Selecting fields there costs nothing and keeps the model's context small:

def main(response: dict) -> dict:
    places = (response.get("local_results") or {}).get("places") or []
    rows = []
    for place in places:
        contact = (place.get("phone") or "").strip()
        digits = sum(character.isdigit() for character in contact)
        rows.append({
            "name": (place.get("title") or "").strip(),
            "category": (place.get("type") or "").strip(),
            "rating": place.get("rating"),
            "reviews": place.get("reviews") or 0,
            "phone": contact if digits >= 10 else None,
            "note": None if digits >= 10 else contact,
            "address": (place.get("address") or "").strip(),
        })
    return {"rows": rows, "count": len(rows)}


# Local check against a live response. Leave everything below out of the Code node.
if __name__ == "__main__":
    import json, os, urllib.request

    body = json.dumps({
        "actor": "scraper.google.search",
        "input": {"q": "plumbers in Austin, TX", "tbm": "lcl"},
    }).encode()
    call = urllib.request.Request(
        "https://api.scrapeless.com/api/v1/scraper/request",
        data=body,
        headers={"Content-Type": "application/json",
                 "x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    )
    with urllib.request.urlopen(call, timeout=180) as reply:
        cleaned = main(json.load(reply))

    print(cleaned["count"], "rows")
    print(json.dumps(cleaned["rows"][0], ensure_ascii=False))

The block above doubles as a local check: run it with your key in the environment and it fetches one live local pack, applies the same function, and prints the first cleaned row. Note that a direct call sends input as an object — the API answers a string with 400 {"message":"invalid input body"}. Dify converts the string form for you on the way out, which is why the same value works in both places.

The cleaning turns 20 raw records into 20 usable ones: names and categories without stray whitespace, a real number in phone when the field holds one, and opening-hours text moved to note instead of being written into a phone column.

For the Amazon shape the same node is a one-liner — return {"product": response["result"]} — and it drops 99% of the payload.

Attach It to an Agent or a Workflow

Both surfaces use the same saved tool, and the choice is about who picks the parameters.

In an Agent, the model decides when to call scraperRequest and what to put in actor and input. That works when instructions name the tool and the data condition explicitly:

When a question depends on current web content, call scraperRequest with
actor "scraper.google.search" and input {"q": "<the search terms>"}, read the
organic_results, and answer from those. Do not answer from memory when the
question is about current prices, rankings, or availability.

In a Workflow, you pin actor on the Tool node and let an upstream node supply only the query. Because input is a string parameter, the reliable pattern is a Code node that builds the JSON text:

def main(query: str) -> dict:
    import json
    return {"payload": json.dumps({"q": query, "tbm": "lcl"})}

Wire payload into the Tool node's input field. Dify's own tools documentation covers the surrounding node wiring in more depth.

If You Self-Host Dify

Self-hosted instances route tool HTTP through a dedicated ssrf_proxy container rather than letting the API container reach the internet directly. When that service is not running, tool calls fail with a DNS error — [Errno -3] Temporary failure in name resolution — which reads like a broken URL rather than a missing container. Bring up the full compose stack, not just api and web, and the same tool works identically to Cloud.

The behavior in this guide was measured on a self-hosted 1.16.1 instance: the schema parsed to one tool, the credential test returned 15,648 bytes of SERP JSON with Custom as the prefix and a 401 string with Basic, and the saved provider listed scraperRequest as an attachable tool.

Conclusion

The Marketplace plugin covers one query string. A custom tool covers the endpoint behind it, which is what a lead flow needs once it starts reading local packs, paging through them, and cleaning the fields before they land anywhere.

The setup cost is one OpenAPI file and four authentication fields — two of which Dify fills in wrong by default. Get those right and every actor in the family becomes available to every app in the workspace, with a Code node doing the shaping that keeps payloads small and columns clean.

Ready to wire it up? Start with a free Scrapeless account, grab your API key, and paste the schema above into your workspace. Usage and plan limits are listed on the Scrapeless pricing page.

FAQ

Q: Should I use the Deep SerpApi plugin or a custom tool?

Use the plugin when a plain Google query is all you need — it exposes one tool with a single query parameter and takes two clicks to install. Use a custom tool when you need the local pack, a page offset, an Amazon listing, or any other actor, because those parameters are not reachable through that single field.

Q: Why does my Dify custom tool return 401 when the same key works in curl?

Two Dify defaults send the key in a form the API does not read. The header name defaults to Authorization instead of x-api-token, and the header prefix defaults to Basic, which makes Dify send x-api-token: Basic <key>. Set the header name to x-api-token and the prefix to Custom.

Q: Why is the input field a string instead of an object?

Dify flattens nested request-body properties when it parses an OpenAPI document, so a nested object becomes a string parameter. Dify accepts either form and normalizes it before the request goes out, so a Code node emitting json.dumps(...) is the dependable way to build it in a Workflow. A direct call to the endpoint is stricter and requires an object.

Q: Does this work on Dify Cloud as well as self-hosted?

Yes. The custom tool is an OpenAPI document plus credentials, with nothing to install on either. Self-hosted instances have one extra requirement: the ssrf_proxy container must be running, because tool HTTP egress is routed through it.

Q: How many results does one request return?

A web search returned eight organic results in the capture used for this guide, and result counts vary by query. The local pack returns 20 places per request, and start: 20 fetches the next page; consecutive pages of one query overlapped slightly, so deduplicate on write.

Q: How do I keep the Amazon response from flooding the model's context?

Select result in a Code node placed after the Tool node. A product call returned 2,226,755 bytes, of which 1,960,588 were the raw html field and only 4,608 were the parsed product, so returning {"product": response["result"]} keeps everything useful and drops the rest.

Q: Can one custom tool cover several actors?

Yes, and that is the point of the design. The endpoint takes actor plus input, so a single scraperRequest operation reaches every actor your account has access to. Adding one to the enum in the schema exposes it in the request builder without a second tool.

Q: Where should the API key live?

In the tool provider's credential field, which Dify stores as a secret and injects at call time. Keeping it there rather than in a node parameter means an exported workflow or a duplicated app does not carry the key with it.