C# Web Scraping: A Practical Guide

TL;DR:

• HttpClient plus AngleSharp is the modern C# pairing. AngleSharp takes CSS selectors, so doc.QuerySelectorAll("div.quote") behaves the way the same selector behaves in a browser console.
• The Scrapeless Universal Scraping API answers with a JSON envelope. The markup arrives inside data, so you read the JSON first and parse HTML second.
• PuppeteerSharp's usual first step downloads a browser. Connecting to a remote one skips it entirely. Every quickstart opens with BrowserFetcher().DownloadAsync(); Puppeteer.ConnectAsync never calls it. Measured: the verification project finished at 7.6 MB with no Chromium on disk and no PuppeteerSharp browser cache.
• The remote CDP path works without ceremony. ConnectAsync accepted a wss:// endpoint carrying a query-string credential and returned title=Quotes to Scrape, quotes on page=10.
• Start free: the Scrapeless dashboard issues a key that works with every example below.

What You Need

Everything below ran on .NET SDK 8.0.129 against quotes.toscrape.com, a site published for scraping practice.

dotnet new console -o scraper
cd scraper
dotnet add package AngleSharp        # 1.7.1
dotnet add package PuppeteerSharp    # 25.5.0

HttpClient and System.Text.Json are in the base class library, so the only dependencies are the parser and, later, the browser driver.

On parser choice: HtmlAgilityPack is the name most C# scraping material reaches for, and it works. AngleSharp is worth preferring for new code because it implements the W3C DOM and takes CSS selectors directly, which means selectors you copy out of browser devtools work unchanged instead of needing translation to XPath.

Fetching and Parsing

using AngleSharp.Html.Parser;
using System.Text;

var http = new HttpClient();
http.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (compatible; csharp-guide/1.0)");

var res  = await http.GetAsync("https://quotes.toscrape.com/");
var html = await res.Content.ReadAsStringAsync();
Console.WriteLine($"http={(int)res.StatusCode} bytes={Encoding.UTF8.GetByteCount(html)}");

var parser = new HtmlParser();
var doc    = await parser.ParseDocumentAsync(html);
var quotes = doc.QuerySelectorAll("div.quote");

Console.WriteLine($"quotes={quotes.Length}");
Console.WriteLine($"first_author={quotes[0].QuerySelector("small.author")!.TextContent}");
Console.WriteLine($"first_text={quotes[0].QuerySelector("span.text")!.TextContent[..40]}");

http=200 bytes=11064
quotes=10
first_author=Albert Einstein
first_text=“The world as we have created it is a pr

Three details are worth calling out.

Set a user agent. HttpClient sends none by default, and a request with no user agent is one of the cheapest things for a site to treat differently.

Encoding.UTF8.GetByteCount(html) is not html.Length. .NET strings are UTF-16, so Length counts UTF-16 code units, not bytes on the wire. On this page the two differ because the quotations use curly quotation marks. Report whichever you mean, but do not conflate them when comparing a direct fetch against an API response.

Scope child queries to the row. quotes[0].QuerySelector(...) searches inside that element; calling doc.QuerySelector(...) inside a loop returns the first quote's value on every iteration, producing a dataset that looks complete and is uniformly wrong.

Where Plain C# Stops

The script works because the target renders server-side and does not screen callers. Two things end that: content that only exists after JavaScript executes, and sites that decline to serve a bare HTTP client. Neither is a .NET limitation — no HTTP client in any language handles either.

From here the escalations are different tools rather than better ones, and the plain path stays cheapest where it works. For the general shape of what a browser adds, the browser automation explainer covers the category.

Escalation One: The Universal Scraping API

This keeps your code an ordinary HttpClient caller and moves the difficulty server-side.

using System.Text;
using System.Text.Json;
using AngleSharp.Html.Parser;

var key = Environment.GetEnvironmentVariable("SCRAPELESS_API_KEY")!;

var payload = JsonSerializer.Serialize(new
{
    actor = "unlocker.webunlocker",
    input = new { url = "https://quotes.toscrape.com/", js_render = false }
});

using var req = new HttpRequestMessage(
    HttpMethod.Post, "https://api.scrapeless.com/api/v2/unlocker/request")
{
    Content = new StringContent(payload, Encoding.UTF8, "application/json")
};
req.Headers.Add("x-api-token", key);

var apiRes = await http.SendAsync(req);
using var json = JsonDocument.Parse(await apiRes.Content.ReadAsStringAsync());

var keys = string.Join(",", json.RootElement.EnumerateObject().Select(p => p.Name));
Console.WriteLine($"http={(int)apiRes.StatusCode} envelope_keys={keys}");

var apiHtml = json.RootElement.GetProperty("data").GetString()!;
var apiDoc  = await new HtmlParser().ParseDocumentAsync(apiHtml);
Console.WriteLine($"quotes={apiDoc.QuerySelectorAll("div.quote").Length}");

http=200 envelope_keys=code,data
quotes=10

The envelope is the part to internalise. The response body is JSON and the markup lives at data. Handing the raw body to ParseDocumentAsync yields a document with no matching elements and throws nothing — selectors quietly return zero results. Read the JSON, take data, then parse, and keep that unwrap in one method.

Note req.Headers.Add("x-api-token", key) rather than an Authorization header. It is a custom header, so it goes on HttpRequestMessage.Headers directly; trying to express it as a typed authentication header is a detour that does not apply here.

Escalation Two: A Remote Browser Without Downloading One

Here is where C# does noticeably better than the usual instructions suggest. Every PuppeteerSharp quickstart begins like this:

// The standard first step — downloads a Chromium build before anything runs.
await new BrowserFetcher().DownloadAsync();
await using var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = true });

Puppeteer.ConnectAsync does not need any of it. You are attaching to a browser that already exists elsewhere, so BrowserFetcher is never involved and nothing lands on disk:

using PuppeteerSharp;

var key = Environment.GetEnvironmentVariable("SCRAPELESS_API_KEY")!;

await using var browser = await Puppeteer.ConnectAsync(new ConnectOptions
{
    BrowserWSEndpoint = $"wss://browser.scrapeless.com/api/v2/browser?token={key}"
});

var page = await browser.NewPageAsync();
await page.GoToAsync("https://quotes.toscrape.com/");

Console.WriteLine($"title={await page.GetTitleAsync()}");
var n = await page.EvaluateExpressionAsync<int>("document.querySelectorAll('.quote').length");
Console.WriteLine($"quotes on page={n}");

title=Quotes to Scrape
quotes on page=10

Two things this buys you beyond convenience.

The project stays small. After running every example in this article, the verification project measured 7.6 MB on disk, contained no Chromium, and left no PuppeteerSharp browser cache. A LaunchAsync workflow adds a browser build to every machine and every container image that runs the code.

EvaluateExpressionAsync<T> deserializes for you. Asking for <int> returns an int, so there is no JsonElement to unwrap and no manual parse between the browser and your variable.

Worth knowing: ConnectAsync accepted a wss:// endpoint carrying its credential in the query string without any additional configuration. That is not universal across languages' CDP clients — some drop the query string or omit TLS server-name negotiation — so if you are porting from another stack, this leg is likely to be easier than the one you left.

Choosing Between the Three

Work down the list rather than up. A page that appears to need a browser very often embeds its data in a <script> tag, and one QuerySelector plus JsonDocument.Parse beats a browser session on every axis.

Conclusion

C# is a comfortable scraping language and the standard library covers more of it than the ecosystem's tutorials imply. HttpClient and AngleSharp handle server-rendered pages with CSS selectors you can paste from devtools, and the two traps worth remembering are counting UTF-16 units when you meant bytes, and querying the document when you meant the row.

The browser story is better than the documentation suggests. The download step in every quickstart is a property of launching a local browser, not of PuppeteerSharp, and ConnectAsync removes it along with the disk footprint — a 7.6 MB project that still drives a real Chrome.

Ready to Scrape With C#?

Create a key on the Scrapeless dashboard and run the HttpClient example against a page you already collect. If it returns what you expect, you are finished — one package, no browser. The Universal Scraping API covers the pages where it does not, and current rates are on the pricing page.

FAQ

Q: AngleSharp or HtmlAgilityPack?

Both parse real-world HTML well. AngleSharp implements the W3C DOM and takes CSS selectors, so devtools selectors port unchanged; HtmlAgilityPack is XPath-first and has a longer history. For new code, AngleSharp usually means less translation work.

Q: Do I have to download Chromium to use PuppeteerSharp?

Only if you launch a local browser. BrowserFetcher().DownloadAsync() belongs to the LaunchAsync path; ConnectAsync attaches to a browser that already exists and never touches it. The verification project here stayed at 7.6 MB with no browser on disk.

Q: Why does my parsing return nothing when the API call succeeded?

You are parsing the envelope. The response is JSON with the markup inside data, so AngleSharp receives a JSON string, matches nothing, and throws nothing. Read data out of the JSON first.

Q: Why does the byte count differ from the string length?

.NET strings are UTF-16, so string.Length counts code units while Encoding.UTF8.GetByteCount counts bytes as transferred. Any page with non-ASCII characters — including typographic quotation marks — will show a difference.

Q: Should I reuse HttpClient?

Yes. Create one and share it, or use IHttpClientFactory in a hosted application. Constructing a new HttpClient per request exhausts sockets under load — Microsoft's own HttpClient guidelines spell out why — and that applies to scrapers exactly as it does to any other .NET HTTP workload.