Selenium Web Scraping: A Practical Python Guide

TL;DR:

• Selenium drives a real Chrome through the WebDriver protocol, so it sees the DOM a page builds with JavaScript — the content a plain HTTP request never receives.
• A live run below reads all ten quotes from a demo page that ships an empty container and fills it client-side; the same page returns zero rows to a bare requests.get.
• Modern Selenium resolves its own driver: Selenium Manager matches ChromeDriver to your installed Chrome, so webdriver.Chrome(options=...) works without a manual download.
• Selenium's real cost is operational — every worker is a full browser exiting from your own IP, which is heavy to scale and straightforward for anti-bot systems to rate-limit.
• The honest-limits pivot sends the same URL to the Scrapeless Universal Scraping API, which renders the page server-side and returns finished HTML, with no browser for you to run or scale.
• Get a Scrapeless API key on the free plan and run both halves yourself.

What Selenium does for scraping

Selenium automates a real browser. Your Python code speaks the WebDriver protocol to ChromeDriver, ChromeDriver drives Chrome, and Chrome does what a browser does: it requests the page, runs the scripts, and builds the DOM. That last part is why scrapers reach for it. A page that assembles its content in the client — a product grid rendered by a framework, a feed that arrives through a background request — is empty in the raw HTML and complete only after the scripts run. Selenium waits for that execution and hands you the finished page.

The tradeoff is weight. Selenium runs an actual Chrome per session, which costs memory and startup time, and the request leaves from your machine's IP. For a handful of pages that is fine. This guide builds the working Selenium scraper first, then shows the point where running your own browsers stops paying off and what to do about it. Every code path here executed against the live page.

Install

Install the Selenium Python bindings:

pip install selenium

You do not install ChromeDriver by hand. Since version 4.6, Selenium ships the Selenium WebDriver tooling with Selenium Manager, which detects your Chrome and resolves the matching driver on first use. You need a recent Google Chrome installed; the bindings handle the rest.

Configure

For the pivot later in this guide, put your Scrapeless API key in the environment so it never lands in source:

export SCRAPELESS_API_KEY="your_scrapeless_api_key"

Basic implementation: render a JavaScript page

Point Selenium at a page that builds its rows in the client, and the elements are there once navigation returns. The demo below ships an empty container and populates it with JavaScript; the raw markup carries no .quote nodes, while the browser-rendered DOM carries ten because Chrome ran the scripts first, following the HTML parsing and scripting model.

import tempfile
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

opts = Options()
opts.add_argument("--headless=new")
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
opts.add_argument(f"--user-data-dir={tempfile.mkdtemp()}")

driver = webdriver.Chrome(options=opts)   # Selenium Manager resolves the driver
try:
    driver.get("https://quotes.toscrape.com/js/")
    quotes = driver.find_elements(By.CSS_SELECTOR, ".quote")
    print("quote blocks on JS page:", len(quotes))
    print("first author:", quotes[0].find_element(By.CSS_SELECTOR, ".author").text)
finally:
    driver.quit()

The run prints the count and the first record:

quote blocks on JS page: 10
first author: Albert Einstein

The --headless=new flag runs Chrome without a visible window, which is what you want on a server. --no-sandbox and --disable-dev-shm-usage keep Chrome stable inside containers and constrained environments, and the throwaway --user-data-dir gives each run its own profile so parallel sessions do not collide.

Advanced patterns: waiting and selecting

driver.get returns when the document loads, but a script-built element may appear a moment later. Rather than sleeping a fixed interval, wait for the specific condition. WebDriverWait paired with expected_conditions blocks until the element you name is present, so the scrape starts the instant the data exists and no sooner:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".quote"))
)

Prefer stable selectors over hashed CSS classes: an id, a data-* attribute, or a semantic tag survives a redesign that renames styling classes. For a page you paginate, read the "next" link's href and follow it rather than guessing page numbers.

Get your API key on the free plan: app.scrapeless.com

The honest limit, and the pivot

Selenium renders JavaScript well; what it does not do is make running browsers cheap or keep your traffic from standing out. Each Selenium worker is a full Chrome, so scaling to many pages means scaling browsers and the memory they need, and every request exits from your own IP, which a busy site can rate-limit or challenge. Selenium has no built-in answer to a managed anti-bot challenge — it renders the page it is given, and a page it never reaches renders as nothing.

The Scrapeless Universal Scraping API takes that operational half off your machine. You send it the URL with js_render enabled; it renders the page server-side behind managed anti-bot handling and rotating residential egress, and returns the finished HTML in its data field. There is no browser for you to launch, and the request does not come from your IP. You parse the returned HTML exactly as you would parse Selenium's page source:

import os, re, json, requests

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"], "Content-Type": "application/json"},
    json={"actor": "unlocker.webunlocker", "input": {"url": "https://quotes.toscrape.com/js/", "js_render": True}},
    timeout=120,
)
resp.raise_for_status()
html = resp.json()["data"]

authors = re.findall(r'<small class="author">(.*?)</small>', html)
print("rendered quote blocks via Universal Scraping API:", html.count('class="quote"'))
print("authors extracted:", len(authors))
print("first author:", authors[0])

The single request returns the same rendered page, server-side:

rendered quote blocks via Universal Scraping API: 10
authors extracted: 10
first author: Albert Einstein

The rule of thumb: keep Selenium while you control the target and the volume is small, and move the fetch to the API when the work is the operations — many pages, blocked IPs, or a challenge Selenium cannot clear. The parsing code does not change; only the source of the HTML does.

Troubleshooting

A driver-version error almost always means Chrome updated and an old driver lingered; on Selenium 4.6 and newer, Selenium Manager resolves the current driver for you, so removing a hand-pinned ChromeDriver from your PATH usually clears it. An empty result on a page you can see in a normal browser means the content arrived after driver.get returned — add the WebDriverWait above for the selector you need. A run that hangs on a page that never goes network-quiet means you waited on the wrong condition; wait for the element, not for the network.

Keep the work inside a site's stated boundaries. Read its terms and its the Robots Exclusion Protocol file, request only public pages, and keep concurrency modest — the same discipline covered in the guide on the Document Object Model that Selenium reads. For a broader tour of discovery and extraction, the walkthrough on building a web scraper from scratch pairs well with this one.

Conclusion

Selenium earns its place when a page needs a real browser to exist: it drives Chrome through the WebDriver protocol, runs the scripts, and hands you the rendered DOM — ten quotes from a page that ships none in its markup. Where it stops paying off is the operations around that browser: the memory of scaling sessions, the exposure of your own IP, and the challenges it cannot clear. At that point the same URL through the Scrapeless Universal Scraping API returns the rendered HTML with no browser to run, and your parsing stays the same. Check the request volume you need against the pricing page before you scale.

Join our community to claim a free plan and compare notes with other developers building scrapers: Discord · Telegram.

FAQ

Q: Do I need to download ChromeDriver to use Selenium?

No. Selenium 4.6 and newer bundle Selenium Manager, which detects your installed Chrome and resolves the matching ChromeDriver on first use, so webdriver.Chrome(options=...) works without a manual driver download.

Q: Why does Selenium find elements that requests cannot?

Selenium runs a real browser that executes the page's JavaScript and builds the DOM, so script-generated elements exist by the time you query them. A bare HTTP client receives only the server's initial markup, which on a client-rendered page contains none of that content.

Q: When should I switch from Selenium to the Universal Scraping API?

Switch when the difficulty is operational rather than rendering — many pages to fetch, your IP getting rate-limited, or a managed anti-bot challenge Selenium cannot clear. The API renders server-side and returns finished HTML, so you keep your parsing and drop the browser fleet.

Q: How do I wait for content that loads after the page?

Use WebDriverWait with expected_conditions to block until a specific selector is present, instead of a fixed time.sleep. The scrape then starts the moment the element exists, which is both faster and more reliable than a guessed delay.

Q: Is scraping with Selenium legal?

Scraping public data is generally permissible, but the responsibility is yours: honor the site's terms and its robots directives, collect only public pages, avoid personal data you have no basis to process, and keep request rates modest.