Pilot
Validate one workflow
Run a representative source sample and inspect both usable results and exceptions. Agree on acceptance criteria before expanding.
Crawl / Documentation Indexing
Turn documentation sites into searchable source material
Crawl selected documentation paths and preserve page context for navigation-aware search and discoverable technical references.
- Choose documentation paths and versions
- Collect content and internal links
- Build and evaluate the downstream search index
Scope
Define the website boundary
Recursive
Collect connected pages
Formats
Choose supported outputs
Sources
Keep page-level provenance
Who uses Documentation Indexing
Crawl selected documentation paths and preserve page context for navigation-aware search and discoverable technical references.
01
Developer experience
Define the business question and review the collected evidence.
02
Documentation teams
Use a repeatable process to compare the observations that matter.
03
Search engineers
Connect reliable inputs to the product or research workflow.
04
Integration owners
Map the source output into your existing application and validate required fields.
05
Workflow operators
Monitor exceptions, maintain collection context and review incomplete runs.
Why this workflow breaks at scale
Design around the failure modes that affect a usable business outcome.
Failure 01
The context gets lost
Mixing old and current documentation can surface the wrong instructions. Preserve version and section context in the search index.
Keep comparable observationsFailure 02
A run ends without a usable result
A completed crawl can still omit important paths or include irrelevant ones. Reconcile discovered pages with scope and coverage expectations.
Validate the actual outcomeFailure 03
The next team cannot verify the result
Preserve source context and reviewable evidence alongside the result. Define who handles incomplete records before production collection.
Make the handoff traceableThe data your workflow can use
Keep the collection layer and the normalized business record clearly defined.
- document_url
- Source or destination reference retained for traceability.
- heading
- Text retained from the relevant source heading.
- version
- Version context associated with the retained document.
- link_targets
- Available source references retained with the observation.
NORMALIZED RECORD · ILLUSTRATIVE
{
"document_url": "https://example.com/public-source",
"heading": "Example public resource",
"version": "v1",
"link_targets": [
"https://example.com/source"
]
}Suggested downstream record, not a literal Scrapeless API response. Field extraction and normalization belong in your application.
Recursive website collection with supported content formats such as Markdown, HTML, links and screenshots.
Example in practice. Index a versioned SDK documentation section with links back to each source page.
How this workflow runs
A focused sequence from approved input to a verified result.
- 01
Choose documentation paths and versions
Start with: Documentation roots, version paths and inclusion rules. Define the minimum usable result before collection.
- 02
Collect content and internal links
Recursive website collection with supported content formats such as Markdown, HTML, links and screenshots. Record failures and partial results separately.
- 03
Build and evaluate the downstream search index
In-scope coverage, duplicate rate and retained document quality. Retain evidence for records that enter the next stage.
Built for the way your team works
Product capabilities support the workflow; your application defines what success means.
- Seed → scope → pages
Collect within a defined scope
Use recursive collection for selected website areas. Start with a small scope and review discovered pages before expanding the crawl.
- Markdown · HTML · links
Choose useful output formats
Crawl supports content formats including Markdown, HTML, links and screenshots. Select the formats required by your consuming application.
- Page → document → source
Keep source identity
Associate each retained document with its URL and capture context. Your pipeline can then maintain versioning and source-linked citations.
- Filter → deduplicate → publish
Prepare content downstream
Apply your own deduplication, content filters and quality checks before indexing or training. Collecting a page is the beginning of dataset preparation.
Plan the cost of your workflow
Crawl documentation describes browser runtime and proxy traffic as cost drivers. Scope, rendering needs and output formats affect consumption.
Try for Free
Production
Measure real consumption
In-scope coverage, duplicate rate and retained document quality.
Scale
Scope the next stage
Bring your workload mix, operating window and observed consumption to discuss the setup that fits your requirements.
These are workload stages, not named subscription plans. Current rates and account limits apply.
Choose how to run the workflow
Compare responsibilities and operating effort using your own representative workload.
Workflow responsibility
With Crawl
Self-managed stack
Manual process
Execution
Scoped website collection
Your team runs and maintains the execution layer.
An operator repeats each collection or browser step.
Business rules
Source permissions, scope, content filtering, versioning, indexing and downstream model use.
Implemented in your application.
Applied by the researcher or operator.
Acceptance
Keep outcome and content checks in your workflow.
Build validation and exception handling.
Review the captured evidence.
Cost planning
Measure product consumption plus downstream processing.
Measure infrastructure and maintenance effort.
Account for time spent per completed task.
Connect it to the stack you already run
Keep business logic and analysis close to the systems that consume the result.
Python
Node.js
Go
Your search or RAG pipeline
Source permissions, scope, content filtering, versioning, indexing and downstream model use. Open the product documentation for current setup and supported options.
Explore more Crawl workflows
Related workflows using the same product, each with its own inputs and acceptance criteria.
AI Training Data
Crawl approved sites into source-linked content that your pipeline can filter, deduplicate and prepare for model training.
RAG Knowledge Data
Crawl selected websites into content your retrieval pipeline can chunk, embed and refresh with traceable citations.
Enterprise Knowledge Bases
Collect scoped company documentation and public resources, then review and organize the content around your team's information needs.
Content Migration & Site Capture
Crawl approved site sections to capture page content, links and metadata for migration mapping and post-migration checks.
Whole-site Change Monitoring
Repeat a scoped crawl and compare normalized content snapshots to detect additions, removals and meaningful page changes.
You are here
Documentation Indexing
Crawl selected documentation paths and preserve page context for navigation-aware search and discoverable technical references.
Market Intelligence Corpora
Crawl approved company and industry sites to give analysts a comparable body of source-linked material for recurring research.
Multimodal Training Data
Collect screenshots alongside page content to build traceable visual-text records for multimodal research and evaluation.
Looking for more Crawl workflows?
Explore more use cases across training data, RAG, documentation search, and website research.
Questions teams ask before they build
Scope the first run with clear expectations.
What do we need to get started?
What does Crawl provide?
Which parts do we build ourselves?
What can make this workflow misleading?
What boundaries should we plan around?
How should we evaluate a pilot?
Put Crawl to work for your team
Bring your sources, expected crawl jobs and acceptance criteria. Start with a focused pilot, then expand on evidence.