How to Run Crawl4AI on an Undetected Fingerprint Browser

TL;DR:

• Crawl4AI already speaks the Chrome DevTools Protocol, so pointing it at Scrapeless is a two-line change. Set browser_mode="cdp" and cdp_url to the Scrapeless WebSocket endpoint, and every arun() crawl runs on a cloud browser instead of local Chromium.
• A local Crawl4AI browser leaks automation on three axes: the navigator.webdriver flag, a headless-default fingerprint, and your own exit IP. Anti-bot systems score all three together.
• The Scrapeless Scraping Browser answers all three server-side — real self-developed Chromium, a consistent per-session fingerprint with no webdriver tell, and residential egress in 195+ countries.
• Your crawler logic is unchanged. Extraction strategies, CrawlerRunConfig, chunking, and markdown output all work exactly as they do locally; only the browser source moves.
• Verified live: a Crawl4AI crawl over the Scrapeless cloud browser returned HTTP 200 and over 11,000 characters of clean markdown, and a separate check read navigator.webdriver as false on a US residential IP.
• Free to start. New Scrapeless accounts include free Scraping Browser runtime — sign up at app.scrapeless.com.

Introduction: give Crawl4AI a browser that reads clean

Crawl4AI drives a real Chromium through Playwright to turn pages into LLM-ready markdown. Run it against a browser on your own machine and every crawl inherits the signals that make automation obvious: it announces itself through the navigator.webdriver property, ships headless-default fonts and canvas behavior, and exits from an IP that reputation services already recognize.

You can patch each of those locally, then keep patching them as Chromium and the detectors move. There is a shorter path. Crawl4AI can attach to any browser that speaks the Chrome DevTools Protocol through its cdp_url setting, and the Scrapeless Scraping Browser is a CDP endpoint reached over one WebSocket URL. Point Crawl4AI at it and the fingerprint, behavioral surface, and exit IP move server-side — the crawler code stays where it is.

What you can do with it

• Crawl anti-bot-protected pages — the cloud browser clears challenges during render, so arun() returns content instead of an interstitial.
• Localize the crawl — pin proxyCountry so a page resolves the way a visitor in that market sees it.
• Send a consistent fingerprint — pass a fingerprint object so user agent, platform, screen, and timezone stay coherent across runs.
• Reuse login state — carry a profileId so cookies and local storage persist between crawls.
• Scale past your laptop — sessions run in the cloud, so a batch of URLs isn't bound to one local Chromium.
• Keep the extraction unchanged — CSS/XPath strategies, LLM extraction, and markdown generation behave identically.

Why Scrapeless Scraping Browser

The Scrapeless Scraping Browser is a customizable, anti-detection cloud browser designed for web crawlers and AI agents. For Crawl4AI specifically, it brings:

• Real self-developed Chromium that runs page JavaScript exactly like Chrome, so client-rendered content attaches before arun() reads it.
• A consistent per-session fingerprint — no navigator.webdriver tell, no headless defaults leaking through.
• Residential egress in 195+ countries, selected per session with one proxyCountry value.
• Session persistence through profileId, so an authenticated crawl keeps its state.
• One connection surface — every option is a query parameter on the same WebSocket endpoint.

Get your API key on the free plan at app.scrapeless.com.

Prerequisites

• Python 3.10 or newer with asyncio
• Crawl4AI installed (pip install crawl4ai)
• A Scrapeless account and API token — sign up at app.scrapeless.com

The examples below are verified against a live Scrapeless session. Full connection details live in the Crawl4AI integration docs.

Install

Crawl4AI brings its own Playwright/CDP support, so the integration needs no extra browser packages.

pip install crawl4ai python-dotenv
export SCRAPELESS_TOKEN=your_api_token_here   # free key at https://app.scrapeless.com

Configure the connection

The Scrapeless endpoint is wss://browser.scrapeless.com/api/v2/browser, and every option is a query parameter. Build the URL once and hand it to BrowserConfig.

import os
from urllib.parse import urlencode

def scrapeless_cdp_url() -> str:
    params = {
        "token": os.environ["SCRAPELESS_TOKEN"],
        "sessionName": "Crawl4AI",
        "sessionTTL": 180000,      # milliseconds
        "proxyCountry": "US",
    }
    return f"wss://browser.scrapeless.com/api/v2/browser?{urlencode(params)}"

Basic crawl over the cloud browser

Set browser_mode="cdp" and pass cdp_url. Everything else — the crawler lifecycle, CrawlerRunConfig, the markdown result — is standard Crawl4AI.

import asyncio
import os
from urllib.parse import urlencode
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

def scrapeless_cdp_url() -> str:
    params = {
        "token": os.environ["SCRAPELESS_TOKEN"],
        "sessionName": "Crawl4AI",
        "sessionTTL": 180000,      # milliseconds
        "proxyCountry": "US",
    }
    return f"wss://browser.scrapeless.com/api/v2/browser?{urlencode(params)}"

async def main() -> None:
    async with AsyncWebCrawler(
        config=BrowserConfig(
            headless=True,
            browser_mode="cdp",
            cdp_url=scrapeless_cdp_url(),
        )
    ) as crawler:
        result = await crawler.arun(
            url="https://www.scrapeless.com/en",
            config=CrawlerRunConfig(wait_for="css:body", scan_full_page=True),
        )
        print("status:", result.status_code)
        print("title:", result.metadata.get("title"))
        print("markdown chars:", len(result.markdown.raw_markdown))

asyncio.run(main())

A live run of this crawl returned status: 200, the page title Effortless Web Scraping Toolkit - Scrapeless, and over 11,000 characters of clean markdown — the same result object Crawl4AI produces locally, sourced through the cloud browser.

Get your API key on the free plan: app.scrapeless.com

Send a consistent fingerprint

A fingerprint object keeps the browser's advertised identity coherent — user agent, platform, screen, language, and timezone all agree, which is what anti-bot scoring checks for. JSON-encode it, then pass it as one more query parameter. The W3C WebDriver specification requires conforming automation to set the webdriver flag; the cloud browser does not carry it, so a consistent fingerprint has no contradicting tell beside it.

import json
from urllib.parse import quote, urlencode

def fingerprint_cdp_url() -> str:
    fingerprint = {
        "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
        "platform": "Windows",
        "screen": {"width": 1920, "height": 1080},
        "localization": {"languages": ["en-US", "en"], "timezone": "America/New_York"},
    }
    params = {
        "token": os.environ["SCRAPELESS_TOKEN"],
        "sessionTTL": 180000,
        "fingerprint": quote(json.dumps(fingerprint)),
    }
    return f"wss://browser.scrapeless.com/api/v2/browser?{urlencode(params)}"

Route the crawl through a chosen region

Turn proxyCountry to an ISO country code to select residential egress; a check against an IP-echo endpoint during verification confirmed the exit IP was a US residential address and navigator.webdriver read as false.

params = {
    "token": os.environ["SCRAPELESS_TOKEN"],
    "sessionTTL": 180000,
    "proxyCountry": "DE",   # or "US", "JP", "ANY"
}

Where local Chromium stops

Running Crawl4AI on your own Chromium is fine for open, unprotected pages. The friction starts where the target scores the browser: a client-rendered app that never attaches its content to a headless default, a challenge interstitial that stalls on a datacenter IP, or a login wall that wants a stable identity across visits. Each of those is a fingerprint, behavior, or IP problem — exactly the three the cloud browser handles server-side. Moving the browser to Scrapeless hands those cases to a managed session while your arun() calls and extraction strategies stay identical.

What you get back

The result is the standard Crawl4AI object — status_code, metadata, markdown, links, media, and any structured extraction you configured. A few honest observations from running it over the cloud browser:

• navigator.webdriver is absent, so the first check a site runs reads like a real Chrome.
• sessionTTL is in milliseconds here — Crawl4AI's CDP mode passes the value straight through, so 180000 is three minutes, not 180.
• The exit IP matches proxyCountry — geo-gated pages resolve as a local visitor sees them.
• wait_for still governs readiness — use a CSS condition (css:body, a content selector) so arun() reads the page after the client render attaches, not before.

Conclusion: same crawler, cleaner browser

Crawl4AI decides what to extract; Scrapeless decides how the browser looks to the site. The integration is two settings — browser_mode="cdp" and a cdp_url — and the rest of your pipeline is untouched. Pin proxyCountry to a residential region, pass a coherent fingerprint, and reuse a profileId for authenticated crawls. For wiring another Python scraping library to the same cloud browser, see the Scrapling production-scraper guide, and compare plans on the Scrapeless pricing page.

Ready to Build Your AI-Powered Crawling Pipeline?

Join our community to claim a free plan and connect with developers building Crawl4AI pipelines: Discord · Telegram.

Sign up at app.scrapeless.com for free Scraping Browser runtime and point Crawl4AI's cdp_url at the cloud browser your targets can't fingerprint.

FAQ

Q: Do I have to change my Crawl4AI extraction code?
No. You change how the browser is created — browser_mode="cdp" plus a cdp_url — and the crawler lifecycle, CrawlerRunConfig, extraction strategies, and markdown output stay exactly the same.

Q: Why not just run Crawl4AI on local Chromium?
Local Chromium runs on your IP, announces automation through navigator.webdriver, and needs you to maintain stealth patches. The cloud browser is a managed session with bundled residential egress and a consistent fingerprint — same crawler API, none of the upkeep.

Q: Do I need a proxy?
No. Residential egress is built in — set proxyCountry to a region or ANY. There is no separate proxy to configure.

Q: Is sessionTTL in seconds or milliseconds?
In Crawl4AI's CDP mode the value is passed through as milliseconds, so 180000 means three minutes. Set it long enough to cover the full crawl.

Q: Is web scraping with Crawl4AI legal?
Accessing publicly available data is generally permitted, but the rules vary by jurisdiction and site. Review the target's terms of service, respect robots directives, and consult counsel for anything sensitive.