PHP Web Scraping: A Practical Guide

TL;DR:

• PHP ships everything a basic scraper needs. ext-curl fetches, DOMDocument plus DOMXPath parse, and no Composer package is involved. A working extraction is about fifteen lines.
• A default php-cli install is not enough. On a clean Ubuntu box php-cli exposed only json and libxml; curl, dom and mbstring each had to be installed separately. Check before you write code, not after.
• DOMDocument::loadHTML() will flood your output with warnings. Real pages are HTML5 and the parser expects HTML4. Wrap the load in libxml_use_internal_errors(true) or the noise reads like a failure when nothing failed.
• The Scrapeless Universal Scraping API returns a JSON envelope. The markup sits in data, so you decode first and parse second.
• chrome-php drives a remote browser, but its default timeout is sized for a local one. Measured against a cloud CDP endpoint, the library's 5-second default completed 2 of 6 attempts; raising sendSyncDefaultTimeout took that to 5 of 6.
• Start free: the Scrapeless dashboard issues a key that works with every example below.

What You Need

Every example here was run on PHP 8.3.6 (cli) against quotes.toscrape.com, a site published specifically for scraping practice.

The first surprise on a clean machine is how little a base PHP install includes. A freshly installed php-cli reported only json and libxml from the set that matters here. The three you actually need arrive separately:

# Ubuntu/Debian — php-cli alone is not enough
apt-get install -y php-cli php-curl php-xml php-mbstring

php -m | grep -E '^(curl|dom|libxml|mbstring)$'
# curl
# dom
# libxml
# mbstring

php-xml is the package that provides DOMDocument; the extension is called dom, which is why grepping for xml finds nothing and sends people in circles.

Fetching a Page With ext-curl

file_get_contents() works for trivial cases, but it gives you no status code, no header control and no timeout worth the name. ext-curl is the baseline for anything real:

<?php
$ch = curl_init('https://quotes.toscrape.com/');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_TIMEOUT        => 30,
    CURLOPT_USERAGENT      => 'Mozilla/5.0 (compatible; php-guide/1.0)',
]);
$html   = curl_exec($ch);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

echo "http={$status} bytes=" . strlen($html) . "\n";

That prints http=200 bytes=11064. Two options carry more weight than they look: CURLOPT_RETURNTRANSFER is what makes curl_exec() return the body instead of printing it, and without CURLOPT_USERAGENT many sites answer a bare PHP client differently or not at all.

Parsing With DOMDocument and DOMXPath

PHP's DOM extension is a full implementation of the W3C DOM standard, and DOMXPath gives you the same query power as any dedicated scraping library. The trap is the loader.

<?php
$doc = new DOMDocument();
libxml_use_internal_errors(true);   // without this, every HTML5 tag warns
$doc->loadHTML($html);
libxml_clear_errors();

$xpath  = new DOMXPath($doc);
$quotes = $xpath->query("//div[@class='quote']");
echo "quotes=" . $quotes->length . "\n";

$first  = $quotes->item(0);
$text   = trim($xpath->query(".//span[@class='text']", $first)->item(0)->textContent);
$author = trim($xpath->query(".//small[@class='author']", $first)->item(0)->textContent);

echo "first_author={$author}\n";
echo "first_text=" . mb_substr($text, 0, 40) . "\n";

Output:

quotes=10
first_author=Albert Einstein
first_text=“The world as we have created it is a pr

Three things are worth pulling out.

libxml_use_internal_errors(true) is not optional in practice. DOMDocument implements HTML4 parsing, so every HTML5 element and unquoted attribute on a modern page raises a warning. Skip the call and a successful scrape buries its own output under parser noise — the PHP manual documents this switch as the supported way to take control of that reporting.

The second argument to DOMXPath::query() scopes the query to a node. Without it, .//span[@class='text'] searches the whole document and you get the first quote's text for every row. That single argument is the difference between per-row extraction and a subtly wrong dataset.

mb_substr() rather than substr() matters because the page uses curly quotation marks. substr() cuts on bytes and will split a multi-byte character into invalid UTF-8, which is exactly the sort of corruption that surfaces three steps later in a database.

Where Plain PHP Runs Out

The script above works because the target renders its content server-side and does not care who is asking. Two things end that: content that only exists after JavaScript executes, and sites that decide a bare HTTP client is not a browser. Neither is a PHP problem — no HTTP client solves either, in any language.

Routing requests through a proxy pool handles a share of the second case, and if you are working inside a framework the Laravel proxy integration guide covers that configuration in more depth than this article does.

At that point there are two escalations, and they are different tools rather than better versions of each other. Keep the plain HTTP path where it works; it stays the fastest and cheapest option by a wide margin.

Escalation One: The Universal Scraping API

The first escalation keeps your code shaped like an HTTP client and moves the hard part server-side. The request is ordinary ext-curl; the response is where the difference shows.

<?php
$key     = getenv('SCRAPELESS_API_KEY');
$payload = json_encode([
    'actor' => 'unlocker.webunlocker',
    'input' => ['url' => 'https://quotes.toscrape.com/', 'js_render' => false],
]);

$ch = curl_init('https://api.scrapeless.com/api/v2/unlocker/request');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_POSTFIELDS     => $payload,
    CURLOPT_TIMEOUT        => 90,
    CURLOPT_HTTPHEADER     => ['Content-Type: application/json', "x-api-token: {$key}"],
]);
$raw    = curl_exec($ch);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

$body = json_decode($raw, true);
echo "http={$status} envelope_keys=" . implode(',', array_keys($body)) . "\n";

$apiHtml = $body['data'];              // the markup lives here, not in $raw
echo "bytes=" . strlen($apiHtml) . "\n";

Output:

http=200 envelope_keys=code,data
bytes=11064

The envelope is the thing to internalise. $raw is JSON, and passing it to DOMDocument produces a document with no matching nodes and no error — selectors simply return zero rows. Decode, take data, then parse. Keep that unwrap in one helper rather than repeating json_decode(...)['data'] across call sites.

Byte count aside, the parsing code is unchanged. Feeding $apiHtml into the same DOMXPath block returns the same ten quotes, which is the point: only the fetch changes.

Escalation Two: A Real Browser From PHP

When content genuinely requires JavaScript, you need a browser. chrome-php/chrome speaks the Chrome DevTools Protocol and can either launch a local Chrome or attach to a remote one over a WebSocket.

composer require chrome-php/chrome
# Using version ^1.16 for chrome-php/chrome  → v1.16.1

<?php
require __DIR__ . '/vendor/autoload.php';

use HeadlessChromium\BrowserFactory;

$key = getenv('SCRAPELESS_API_KEY');
$uri = "wss://browser.scrapeless.com/api/v2/browser?token={$key}";

$browser = BrowserFactory::connectToBrowser($uri, [
    'sendSyncDefaultTimeout' => 60000,   // see below — the default is 5000
]);

$page = $browser->createPage();
$page->navigate('https://quotes.toscrape.com/')->waitForNavigation();

echo "title: "          . $page->evaluate('document.title')->getReturnValue() . "\n";
echo "quotes on page: " . $page->evaluate('document.querySelectorAll(".quote").length')->getReturnValue() . "\n";

$browser->close();

Output:

title: Quotes to Scrape
quotes on page: 10

The Default Timeout Is Sized for a Local Browser

That sendSyncDefaultTimeout line is the part worth the article. chrome-php defaults it to 5000 ms, which is generous for a Chrome running on the same machine and marginal for one across a TLS connection. Two matched sets of six runs, same script, same target, changing only that option:

Two conclusions follow, and the second is the one people miss.

Raising the timeout is necessary. Left at the default, a majority of attempts died on the same message, and it is a message that misleads — it names a timeout, so it reads like the page or the network is slow, when what actually expired is the library's own wait on a protocol round trip.

Raising it is also not sufficient. The one failure that survived was Cannot connect to the browser, make sure it was not closed, raised about five seconds in, while the connection was still being established rather than during a command. A larger command timeout has no bearing on that. Treat obtaining the browser as its own fallible step in the job's design, distinct from the work you do once you hold one, and keep $browser->close() in a finally so a mid-job failure never strands a session.

One Diagnosis That Was Wrong

The first theory for those timeouts was that the URI's query string never reached the WebSocket handshake, taking the ?token= with it. There is real evidence for it: Protocol::validateSocketUri() returns only [$scheme, $host, $port] and discards path and query outright.

It is still wrong. The handshake is built elsewhere, by Protocol::getRequestHandshake(), which calls a separate validateUri() returning five elements and explicitly re-appends the query before the request line. The token does arrive. The failure was latency, and the two functions parsing the same URI to different depths is a coincidence that looks exactly like a bug.

It is a useful reminder that in a library with more than one URI parser, finding one that drops your data does not mean it is the one on the path you care about.

Choosing Between the Three

Work down that list, not up. Most pages that look like they need a browser turn out to embed their data in a <script> tag, and a DOMXPath query plus json_decode() beats a browser session on every axis.

Conclusion

PHP is a perfectly reasonable scraping language, and the parts people expect to be missing are in the standard library. ext-curl and DOMXPath cover server-rendered pages completely, with libxml_use_internal_errors(true) and a scoped second argument to query() as the two details that separate working code from quietly wrong code.

When a target stops cooperating, escalate deliberately. The API path keeps your code an HTTP client and asks only that you unwrap an envelope. The browser path costs a session and, if that browser is remote, one specific piece of configuration: chrome-php's five-second default is a local-Chrome assumption, and leaving it in place cost four of six connections here.

Ready to Scrape From PHP?

Create a key on the Scrapeless dashboard and run the ext-curl example against one page you already collect. If it returns what you expect, you are done — no dependency, no browser. The Universal Scraping API is there for the pages where it does not, and current rates are on the pricing page.

FAQ

Q: Do I need Composer to scrape with PHP?

No. Fetching with ext-curl and parsing with DOMDocument and DOMXPath needs nothing beyond the extensions in a standard install. Composer only enters the picture for a browser driver such as chrome-php/chrome.

Q: Why does DOMDocument print warnings on every page?

Because it implements HTML4 parsing and modern pages are HTML5. The warnings are informational rather than failures. Call libxml_use_internal_errors(true) before loadHTML() and libxml_clear_errors() after, and inspect libxml_get_errors() when you actually want to see them.

Q: Why does my parsing return nothing when the API request succeeded?

You are almost certainly parsing the envelope. The Universal Scraping API returns JSON with the markup inside data, so DOMDocument receives a JSON string and finds no matching nodes without raising an error. Decode the response and parse data.

Q: Which XPath do I use for a class attribute?

//div[@class='quote'] matches only an exact attribute value. For an element carrying several classes, use //div[contains(concat(' ', normalize-space(@class), ' '), ' quote ')], which avoids matching quote-footer the way a bare contains() would.

Q: Should I use PHP for a large scraping project?

For fetch-and-parse work at scale, yes — curl_multi_* gives you real concurrency and the DOM extension is fast. The place PHP is weaker is long-lived browser automation, where the tooling around Playwright and Puppeteer is more mature. A common split is PHP for the HTTP path and a browser service for the pages that genuinely need one.