What Is Crawlee?

Scrapeless Agent Browser provides remote browser infrastructure for automation workflows that need JavaScript rendering.

Crawlee is an open-source crawling and web scraping library with JavaScript and Python implementations. It organizes tasks such as processing URLs, choosing HTTP or browser-based execution, extracting records, and saving results. A crawler framework coordinates work across pages; it is different from both the browser engine and the infrastructure that hosts that browser.

The distinction becomes useful as soon as a script has more work than one page visit. You need to know which URLs belong to the job, which have already been handled, and where each result should go. Crawlee supplies components for that orchestration, while your handlers still decide what a valid record looks like.

What Crawlee Adds Above a Page Script

Crawlee adds crawl orchestration around the code that fetches or opens a page. Its crawler framework architecture includes request storage, result storage, browser management, and resource-aware scheduling. The exact classes and options depend on the language implementation and release you use.

A standalone script might open a URL and print a title. A crawl must also manage the surrounding decisions: where the URL came from, whether it should be visited, what handler should process it, and how its result relates to the rest of the dataset. Those concerns remain even when the page itself is easy to parse.

Do not interpret framework automation as a complete extraction specification. Crawlee does not know which price belongs to the product you intend to measure or whether an empty result is valid. Your application needs explicit selectors or parsing rules, required fields, and a completion condition that follows the data task.

The Request Queue Defines the Frontier

A request queue represents the work waiting to be processed and helps a crawl track its progress. It is the frontier between discovered URLs and completed handling, not the final dataset.

URL identity deserves care. Different URLs can identify the same content when tracking parameters or alternate routes are involved. Conversely, a query parameter may select a different language, category, or page of results. Establish which parts of a URL change the requested content before you normalize or deduplicate it.

Scope discovery separately from extraction. A category page can discover product links, while product handlers collect the detailed records. Limit discovery to intended hosts and paths so navigation links do not pull unrelated account, support, or search pages into the job.

For a bounded catalog crawl, define an allowed URL family and a maximum job scope. When the queue contains no more permitted work, evaluate output completeness against the intended inventory. An exhausted queue alone does not prove completeness if discovery rules excluded valid pages.

HTTP Crawling and Browser Crawling Have Different Costs

HTTP crawling fits content present in a response, while browser crawling fits content that requires page execution or interaction. Start by inspecting how the required information reaches the page.

An HTTP crawler can acquire HTML and parse it without creating a browser. If the desired values are present in that response, this avoids rendering work. If the application builds its records through JavaScript after load, a browser may be necessary to reproduce the visible state.

The HTTP request and response semantics define transport behavior, but a successful response is not a promise of usable records. Validate the content type, expected page identity, and required fields before accepting the result.

A mixed crawl may use different handlers for different page types. Keep the output contract consistent where the records represent the same entity, but preserve provenance so you can explain which path produced each value.

Request Handlers Translate Pages Into Records

A request handler is the application-specific step that interprets a response or browser page. It should connect page identity to the record you store.

For a product task, begin by confirming that the final page still represents the intended product. Then locate the relevant fields and decide how to represent missing values. A displayed zero, an absent price, and a parse failure should not silently become the same value.

Design handlers around page roles. Listing pages discover details; detail pages extract entities; supporting pages may provide metadata. Distinct roles make it easier to see whether a job failed during discovery or during extraction.

Wait for meaningful application conditions in browser handlers. A navigation event can occur before a record is rendered, and a visible container can exist while its contents are still loading. The condition should reflect the data that the handler is about to read.

Dataset Storage Is Separate From Crawl Progress

Result storage holds extracted output, while request storage tracks work. Keeping those responsibilities distinct helps you inspect incomplete jobs and avoid mistaking a visited URL for a valid record.

Choose a schema before collecting a large sample. Include a stable entity identifier when one exists, the source URL, and the fields the downstream process actually needs. Store extraction time where freshness matters, and distinguish original values from normalized values if normalization could hide useful context.

A dataset can contain repeated entities when discovery reaches the same item through multiple categories. Treat entity deduplication as its own data decision rather than assuming queue deduplication has solved it. URL uniqueness and business-entity uniqueness are different requirements.

Inspect the stored output directly. A handler that reaches the end without raising an exception may still produce blank fields or the wrong page title. Define acceptance checks for record counts where justified, required-field coverage, and a sample of source-to-record matches.

Sessions and Resource Limits Need Explicit Policies

A crawl needs a state policy and a resource policy because page behavior and browser capacity change how work can be scheduled. Choose those policies around the target workflow rather than a general desire to maximize simultaneous requests.

State affects content. The browser cookie model can preserve language preferences, consent decisions, or account identifiers. Decide whether each task should use a fresh environment or continue an authorized identity, and prevent unrelated jobs from inheriting that identity.

Resource limits should reflect the execution path. Browser pages consume a different mix of memory and CPU from HTML parsing. Measure a representative workload, then set bounded concurrency and job limits that keep output quality observable.

Stop at access boundaries that require permission or human action. A crawl configuration is not authorization to collect restricted information. Keep the crawl within the intended public or otherwise authorized scope, and minimize retained account or personal data.

How Crawlee Relates to Remote Browser Infrastructure

Crawlee coordinates crawling, while a remote browser service hosts browser execution. These layers can be evaluated separately even when a workflow needs both.

Scrapeless Agent Browser provides a managed browser environment, and its Puppeteer connection interface documents one way for a client to control that environment. A Crawlee deployment needs a compatible browser-pool configuration and explicit ownership of the remote connection.

If a proposed architecture uses Crawlee and a hosted browser, verify the specific connection path, context ownership, and cleanup behavior in a separate integration test. The crawler's queue should remain accountable for work, while the browser lifecycle should remain accountable for execution resources.

The separation is also explained in the Node.js managed browser discussion. Review Scrapeless plans against the browser portion of the workload; framework storage and application processing remain their own operational concerns.

Conclusion

Crawlee organizes the work around scraping: discover URLs, process the right page type, extract a defined record, and store the result. The main design decisions are crawl scope, execution mode, state ownership, and output validation. Start with a bounded sample and inspect its records before expanding the crawl.