How to Reduce Claude Code Web Research Tokens With Structured Extraction

TL;DR:

• Claude Code web research tokens are driven by what reaches the model, not only what crosses the network. Raw HTML, repeated navigation, duplicate URLs, tool metadata, and verbose results can all consume context.
• Reduce content before the reasoning step. Prefer main-content extraction, narrow selectors, field-level extraction, and a JSON Schema that rejects incomplete results.
• Keep source URLs and evidence snippets. A smaller payload is useful only if the answer remains auditable and complete.
• MCP can separate acquisition from reasoning. Claude Code chooses a bounded Scrapeless tool; the tool returns only the fields the task requires.
• Our reproducible proxy test cut a fixed input from 181,892 to 434 comparison tokens. It used one public documentation page, one question, and cl100k_base; these are not Claude billing tokens.

Claude Code can search the web, fetch a page, call MCP tools, and reason over the returned material. That makes research convenient, but convenience can hide a basic cost problem: a question that needs six facts may pull tens of thousands of irrelevant characters into context.

The fix is not “summarize more aggressively.” Summarization is another model task and can remove the evidence you needed. A better workflow reduces content at acquisition time, validates the result, and sends the main model a small evidence contract.

This tutorial builds that workflow for Claude Code with Scrapeless MCP Server and Universal Scraping API patterns.

Where Claude Code Web Research Tokens Come From

Treat the research path as four separate volumes:

acquired bytes → extracted characters → model-context tokens → structured answer tokens

They are related, but they are not interchangeable.

Page acquisition volume

This is what the browser, fetcher, or scraping service receives. A rendered page may include scripts, styles, navigation, cookie banners, embedded state, and content for several routes. Acquisition volume affects network and scraping cost, but it does not have to enter Claude's context.

Returned characters

The tool chooses what it returns: raw HTML, readable Markdown, selected elements, or a JSON object. This is the most useful control point. Removing boilerplate here saves every later step from processing it.

Main-model context

Claude Code sees system instructions, conversation history, tool definitions, tool results, and your current request. A concise tool response can still sit beside a large project context. Use the current Claude Code context-window documentation to understand how context is managed, but measure your own workflow rather than assuming one fixed capacity or price.

Structured answer output

A JSON Schema constrains the final answer. It does not automatically shrink the page content that preceded it. Apply structure both to tool results and to the final response.

Establish a Baseline Before Optimizing

Use one research question and a fixed source set. Record:

• source URLs requested;
• characters returned by each tool;
• model and Claude Code version;
• tokenizer or API usage fields used for measurement;
• input and output token counts;
• required answer fields and completeness results.

Claude Code exposes current CLI options with claude --help. On the machine used for this article, Claude Code 2.1.162 listed --mcp-config, --tools, --output-format, and --json-schema. The current Claude Code tools reference documents WebSearch and WebFetch as built-in tools.

Do not confuse those Claude Code tools with Anthropic API web tools. The API's web fetch documentation describes server-side features such as dynamic filtering. A feature documented for the API is not automatically a Claude Code WebFetch option.

Step 1: Prefer Main Content Over Raw HTML

Raw HTML is useful for debugging selectors or preserving exact markup. It is usually a poor research payload.

Ask the acquisition layer to remove:

• script, style, SVG, and template content;
• site navigation and repeated footers;
• consent and account shells;
• hidden state not required by the question;
• unrelated recommendations and comments.

Readable Markdown is often a good first reduction. It retains headings, lists, links, and code while discarding much of the presentation layer. Still validate that the title and required section are present; a clean login page is not a successful extraction.

Step 2: Extract Only the Fields the Question Needs

Main content can remain much larger than the answer. Turn the question into an extraction contract before fetching several pages.

For a documentation comparison, the contract might be:

{
  "type": "object",
  "required": ["source_url", "tools", "complete"],
  "properties": {
    "source_url": { "type": "string", "format": "uri" },
    "tools": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["name", "evidence"],
        "properties": {
          "name": { "type": "string" },
          "evidence": { "type": "string", "maxLength": 1200 }
        }
      }
    },
    "complete": { "type": "boolean" }
  }
}

Keep evidence short, but do not reduce it to an unsupported value. A field such as supports_web: true is compact and hard to audit. The source URL plus a bounded evidence snippet lets a reviewer verify the interpretation.

For repeated page shapes, use targeted selectors or a structured extraction endpoint. For varied pages, request readable content first, then select relevant sections with deterministic code where possible.

Step 3: Deduplicate URLs Before Acquisition

Research agents often encounter the same document through navigation, search parameters, language aliases, or fragments. Normalize before fetching:

1. resolve relative URLs;
2. remove fragments;
3. apply an approved query-parameter policy;
4. follow redirects once and record the final canonical identity;
5. hash accepted content to catch mirrors or repeats.

Do not delete every query parameter. Locale, version, product, or date parameters may change the document. The normalization rule belongs to the source policy, not a universal string cleaner.

A small cache can also prevent repeated collection in one run. Key it by canonical source, locale, extraction contract version, and freshness requirement.

Step 4: Connect Claude Code to Scrapeless MCP

MCP keeps the agent-facing interface small. Claude Code sees named tools and schemas; Scrapeless handles search, public-page scraping, or cloud-browser operations behind them.

Prerequisites:

• Claude Code and Node.js installed;
• a Scrapeless account and API key;
• an authorized public target;
• a page and tool-call budget.

Store the key in an environment variable. This project-level example uses variable expansion so the secret is not committed:

{
  "mcpServers": {
    "scrapeless": {
      "command": "npx",
      "args": ["-y", "scrapeless-mcp-server"],
      "env": {
        "SCRAPELESS_KEY": "${SCRAPELESS_KEY}"
      }
    }
  }
}

Save the object as .mcp.json in the approved project and start Claude Code from that project. Run claude mcp list to inspect connection health. Claude Code requires approval for project-scoped MCP servers; inspect the command and environment keys before approving them.

During editorial verification, the current Scrapeless package was launched over stdio with the standard MCP client flow. A credentialed web call was not run because no production key was present in the verification environment. Treat the first real tool call as an acceptance test: one allowed URL, one required field set, and no broad crawling.

The current Claude Code MCP documentation explains scope, transports, tool discovery, output limits, and environment-variable expansion. The Scrapeless Claude integration guide provides the product-specific server configuration.

Step 5: Give the Agent a Bounded Research Contract

“Research this topic” invites exploration. A bounded prompt defines sources, fields, and a stop condition.

Use a request like this:

Search for up to five official sources about the named product. Deduplicate canonical URLs. For each accepted source, return title, final URL, publication or update date when visible, and one evidence snippet supporting the required feature. Stop after three complete sources. Mark missing fields; do not infer them.

This contract controls four failure modes at once: unlimited search, duplicate acquisition, verbose results, and invented fields.

Also filter the MCP tool set. A documentation task may need search and one-shot Markdown extraction, not every browser action. Fewer visible tools reduce selection ambiguity and simplify permission review.

Step 6: Measure the Reduction and Completeness Together

We used the public Claude Code tools-reference page and one fixed question:

Which built-in Claude Code tools can search or fetch public web content, and what constraints should a research workflow apply?

The same cl100k_base tokenizer counted the question plus each material variant. It is an open comparison proxy, not Anthropic's tokenizer and not a Claude billing measurement.

Targeted JSON used about 95.4% fewer comparison tokens than main content and 99.8% fewer than raw HTML. Those percentages describe this page and this extraction contract only.

The completeness check was deliberately narrow: both required tool names had to exist in the main text and in the extracted object, with source identity preserved. A production evaluation should also score whether each evidence snippet supports the final answer.

If you have Claude API credentials and need billing-accurate input counts, use Anthropic's current token-counting or usage interfaces for the exact model. Keep the model, question, tool schemas, source snapshot, and output contract constant across variants.

Step 7: Add a Result Acceptance Gate

Compression is not correctness. Before content enters the main reasoning step, validate:

• the final URL belongs to the allowlist;
• the page identity matches the requested document;
• required fields exist and have the right types;
• evidence includes the claimed entity or term;
• content is not an error, consent, login, or access-challenge shell;
• the record includes collection time and extraction-contract version;
• the total returned characters remain under the task budget.

Return typed outcomes such as accepted, missing_fields, wrong_page, access_required, or over_budget. Claude can decide whether to stop or use another approved route without treating every failure as ordinary prose.

When to Use Universal Scraping API Instead

MCP is useful when Claude Code should discover and choose a web capability interactively. Universal Scraping API is a better boundary when your program already knows the target and wants a predictable request from CI, a data job, or a service.

Use the API path when you need to:

• call extraction from deterministic application code;
• centralize rate and budget controls outside the agent;
• normalize results before Claude Code runs;
• cache accepted records across many research sessions.

The same principle applies: request the least expensive output that can meet the acceptance contract, then send Claude only the accepted fields. Review the Universal Scraping API product page and current documentation for supported endpoints and request fields.

Common Mistakes

Sending raw pages “just in case”

This shifts selection work to the most context-sensitive part of the system. Preserve raw material outside the prompt and send an evidence bundle instead.

Optimizing tokens without an answer contract

A tiny response that omits a required fact is not efficient. Measure accepted answers per cost, not token reduction alone.

Dropping source identity

Without the final URL and evidence, the result cannot be audited or refreshed safely.

Exposing keys in prompts or committed config

Use environment variables and project secret controls. Never paste a production key into a prompt, example, log, or repository.

Assuming tool output limits guarantee relevance

An output cap prevents unbounded size. It does not select the right passage or validate the page.

Production Checklist

• Fix the question, model, source set, and output schema for the benchmark.
• Count returned characters at every tool boundary.
• Deduplicate canonical URLs before collection.
• Prefer Markdown, selectors, or structured fields over raw HTML.
• Preserve final URL, evidence, collection time, and contract version.
• Reject wrong-page and incomplete results before main-model reasoning.
• Limit visible MCP tools and approved targets.
• Keep secrets outside prompts and version control.
• Compare accepted-result cost, not isolated token counts.

Read the Scrapeless MCP Server overview, inspect Scrapeless pricing, and start with one bounded research task.

FAQ

Q: Does WebFetch always use fewer Claude Code tokens than a browser tool?

No. Token use depends on the material returned to context. A concise browser extraction can be smaller than a verbose fetch, while a clean fetch can be smaller than browser HTML. Measure returned content.

Q: Can JSON Schema reduce input tokens?

It can reduce and validate structured outputs, but it does not automatically shrink page content. Apply a schema at the extraction boundary and again at the final answer if needed.

Q: Are the token counts in this article Claude tokens?

No. They are a reproducible cl100k_base comparison proxy. Use the current Anthropic token-counting or usage interface with your exact Claude model for billing-relevant numbers.

Q: Why use MCP instead of calling a scraping API directly?

Use MCP when Claude Code should discover and invoke a bounded tool during an interactive task. Use the direct API when application code already knows when and how to collect the page.

Q: How do I know extraction did not remove a required fact?

Define required fields and evidence before collection, then run completeness and semantic checks. Keep the source URL and raw snapshot outside the prompt for audit or reprocessing.