Java Web Scraping With jsoup: Fetch, Parse, and Render

TL;DR:

• jsoup is a Java HTML parser with a CSS selector engine, and it ships its own HTTP client, so a working scraper is one dependency and about twenty lines.
• jsoup keeps only the first 2 MiB of a response by default. On a 2,235,648-byte page it kept exactly 2,097,152 bytes, dropped 68 of 255 reference entries, lost an entire section, and threw nothing.
• attr("href") returns the href exactly as written in the HTML. attr("abs:href") resolves it against the document's base URI.
• jsoup does not execute JavaScript. On a client-rendered page the same selector code found 0 items directly and 10 items when the HTML arrived pre-rendered.
• Routing the fetch through the Scrapeless Universal Scraping API changes the transport only — the parsing method is untouched.
• The Scrapeless free plan is enough to run every request in this guide.

Java is where a lot of scraped data ends up. If the service that consumes the data is a Spring or Quarkus application, keeping extraction in the same JVM removes a language boundary, a serialization step, and a second deployment target.

The library that makes that practical is jsoup. It parses real-world HTML the way a browser does — closing unclosed tags, fixing nesting, and normalizing attributes, following the error-handling rules in the HTML parsing specification — and then exposes the result through a CSS selector API.

This guide builds a working scraper against two live sites, then measures two behaviours that decide whether the scraper is correct: what jsoup silently discards on a large page, and what it returns on a page that renders in the browser.

What jsoup Gives You

jsoup is a parser first and an HTTP client second. Jsoup.connect(url) returns a fluent request builder; .get() performs the request and hands back a Document you query with selectors.

String url = "https://books.toscrape.com/";
Document doc = Jsoup.connect(url).get();
String title = doc.selectFirst("h1").text();

That Document is a parsed tree rather than a string, so a malformed page still yields a queryable structure. It also carries the base URI it was fetched from, which is what makes absolute URL resolution possible later on.

What jsoup does not do is run scripts. It has no JavaScript engine and no DOM event loop. Whatever the server sends is what you get to parse.

Set Up the Project

One dependency covers parsing. Gson is here only to read the JSON envelope in the last section; if you skip that section you can drop it.

<dependencies>
  <dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.21.1</version>
  </dependency>
  <dependency>
    <groupId>com.google.code.gson</groupId>
    <artifactId>gson</artifactId>
    <version>2.14.0</version>
  </dependency>
</dependencies>

Pin the compiler plugin explicitly. Maven 3.8.7 still binds maven-compiler-plugin 3.1 by default, which ignores maven.compiler.release and stops with Source option 5 is no longer supported:

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <version>3.15.0</version>
    </plugin>
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>exec-maven-plugin</artifactId>
      <version>3.5.0</version>
      <configuration>
        <mainClass>com.example.Scraper</mainClass>
      </configuration>
    </plugin>
  </plugins>
</build>

With maven.compiler.release set to 17 the code below compiles on any current JDK. The verification run used OpenJDK 21.0.11.

Fetch a Page and Parse It

Set a user agent and a timeout on every request. jsoup's default agent identifies itself as jsoup, and plenty of sites vary their response on that alone.

static final String USER_AGENT =
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    + "(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36";

static Document fetch(String url) throws IOException {
    return Jsoup.connect(url)
        .userAgent(USER_AGENT)
        .timeout(30_000)
        .get();
}

timeout is milliseconds and applies to both the connect and the read. A get() that exceeds it raises SocketTimeoutException; a non-2xx status raises HttpStatusException, which carries the code.

Select the Data

Selectors are standard CSS. select returns every match as an Elements collection; selectFirst returns one Element or null.

record Book(String title, String price, String url) {}

static List<Book> books(Document doc) {
    List<Book> found = new ArrayList<>();
    for (Element card : doc.select("article.product_pod")) {
        Element link = card.selectFirst("h3 > a");
        found.add(new Book(
            link.attr("title"),
            card.selectFirst("p.price_color").text(),
            link.attr("abs:href")));
    }
    return found;
}

Against the live catalogue this returns 20 books, the first being A Light in the Attic at £51.77.

The abs: prefix on that last line is doing real work. The anchor in the source reads:

catalogue/a-light-in-the-attic_1000/index.html

attr("href") gives you that string verbatim. Prefixing the attribute name with abs: resolves it against the document's base URI using the algorithm in the WHATWG URL Standard, producing:

https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html

A scraper that stores the raw value works fine until the day something else tries to fetch it.

The Limit That Costs You Data

jsoup caps the response body it will read. The default is 2 MiB, documented on the jsoup request interface. Past that point it stops reading and parses what it has. It raises no exception, logs no warning, and leaves no flag on the Document to say the body was cut short.

Use execute() instead of get() when you want to see the response before parsing it:

Connection.Response capped = Jsoup.connect(big)
    .userAgent(USER_AGENT).timeout(60_000).execute();
Connection.Response whole = Jsoup.connect(big)
    .userAgent(USER_AGENT).timeout(60_000).maxBodySize(0).execute();

Run against a 2,235,648-byte Wikipedia comparison page, the two responses differ by exactly the cap:

http status, default limit: 200
bytes received, default limit: 2097152
bytes received, maxBodySize(0): 2235648
table rows, default limit: 544
table rows, maxBodySize(0): 544
reference entries, default limit: 187
reference entries, maxBodySize(0): 255
last section, default limit: References
last section, maxBodySize(0): External links

Both runs returned HTTP 200. Both produced a Document. The comparison tables the page exists for came through identically, 544 rows either way, because they sit near the top of the document.

Everything below the cut is simply absent. The reference list lost 68 of its 255 entries, and the final External links section does not exist in the truncated tree at all. A scraper reading the tables would never notice; a scraper collecting citations would silently under-report by a quarter and still look healthy.

maxBodySize(0) removes the limit. Set it deliberately rather than by reflex — an unbounded read on an unexpected response is its own problem — but set it consciously, and compare against the Content-Length the server reports, which the HTTP semantics specification defines as the body's octet count.

When the Page Renders in the Browser

https://quotes.toscrape.com/js/ marks the boundary. It serves its quotes as a JavaScript array and builds the DOM client-side, and jsoup fetches it successfully:

html bytes fetched directly: 5479
quotes found by jsoup:       0

5,479 bytes, HTTP 200, zero results. The markup jsoup received genuinely contains no div.quote elements — they are created after the script runs, and jsoup has no script engine.

Most guides answer this by switching to a browser automation tool, which means rewriting the extraction code too. That trade-off is the same one Cheerio and Puppeteer sit on either side of in Node, and it costs the same thing in Java. The narrower fix is to change only how the HTML arrives. That job belongs to the Scrapeless Universal Scraping API. It renders the page and returns the resulting HTML as a string, which you hand to the same parser.

The JDK's built-in HttpClient is enough — no HTTP dependency required:

static String render(String url) throws IOException, InterruptedException {
    JsonObject input = new JsonObject();
    input.addProperty("url", url);
    input.addProperty("proxy_country", "US");
    input.addProperty("js_render", true);

    JsonObject payload = new JsonObject();
    payload.addProperty("actor", "unlocker.webunlocker");
    payload.add("input", input);

    HttpRequest request = HttpRequest.newBuilder()
        .uri(URI.create("https://api.scrapeless.com/api/v2/unlocker/request"))
        .header("Content-Type", "application/json")
        .header("x-api-token", System.getenv("SCRAPELESS_API_KEY"))
        .timeout(Duration.ofSeconds(180))
        .POST(HttpRequest.BodyPublishers.ofString(payload.toString(), StandardCharsets.UTF_8))
        .build();

    HttpResponse<String> response = HttpClient.newHttpClient()
        .send(request, HttpResponse.BodyHandlers.ofString());
    if (response.statusCode() != 200) {
        throw new IOException("unlocker returned HTTP " + response.statusCode());
    }
    return JsonParser.parseString(response.body())
        .getAsJsonObject().get("data").getAsString();
}

The response envelope is {"code": ..., "data": "<rendered html>"}. Parse that string with Jsoup.parse(html, url) — passing the URL sets the base URI, so abs:href keeps working — and run the identical selector method:

static List<String> quotes(Document doc) {
    List<String> found = new ArrayList<>();
    for (Element quote : doc.select("div.quote")) {
        found.add(quote.selectFirst("span.text").text()
            + " -- " + quote.selectFirst("small.author").text());
    }
    return found;
}

rendered html bytes:            8940
quotes found by the same parser: 10

Same method, same selectors, same Document API. The only thing that changed is where the 8,940 bytes came from. Keep your API key in the environment, as SCRAPELESS_API_KEY above, rather than in the source. The Universal Scraping API getting-started guide covers the remaining request parameters.

Getting started takes a minute — create a free Scrapeless account and the free plan covers everything in this guide.

Run It

With the class assembled, one command compiles and runs it:

export SCRAPELESS_API_KEY="your-api-key"
mvn -q -B compile exec:java

The full output from the verification run:

jsoup 1.21.1 | java 21.0.11
--- static page, parsed directly ---
books on page 1: 20
first title: A Light in the Attic
first price: £51.77
first url:   https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
href as written in the HTML: catalogue/a-light-in-the-attic_1000/index.html
--- default body-size limit ---
http status, default limit: 200
bytes received, default limit: 2097152
bytes received, maxBodySize(0): 2235648
table rows, default limit: 544
table rows, maxBodySize(0): 544
reference entries, default limit: 187
reference entries, maxBodySize(0): 255
last section, default limit: References
last section, maxBodySize(0): External links
--- javascript-rendered page ---
html bytes fetched directly: 5479
quotes found by jsoup:       0
--- same page through the Universal Scraping API ---
rendered html bytes:            8940
quotes found by the same parser: 10

Troubleshooting

Source option 5 is no longer supported. Maven bound its default maven-compiler-plugin rather than yours. Pin the plugin version in <build><plugins> as shown above.

selectFirst returned null and the next line threw NullPointerException. The selector matched nothing. Check the element against the HTML jsoup actually received — doc.html() — not against what the browser inspector shows, which is the post-script DOM.

Counts are lower than the page shows. Compare response.bodyAsBytes().length against the server's Content-Length. If they match at 2,097,152, the body-size cap is the cause.

HttpStatusException: 403. The server rejected the request rather than the parse. Set a realistic user agent first; if the page also requires rendering, the pivot above applies.

Mojibake in the extracted text. jsoup reads the charset from the Content-Type header, then the meta tag. When a server declares neither, pass the encoding explicitly to Jsoup.parse(InputStream, String, String).

Conclusion

For static HTML, jsoup and the JDK are the whole toolchain: one dependency, a fluent request, a CSS selector API, and a parsed tree that survives bad markup. The two behaviours that decide whether the results are trustworthy are both invisible by default — the 2 MiB body cap that returns a healthy-looking partial document, and the empty result set on a client-rendered page.

Both are cheap to check. Compare received bytes against Content-Length, and compare a selector's count against what the page displays. When the second check fails because the markup arrives empty, the fix is to change the transport and leave the parser alone.

Ready to try it? Start with the Scrapeless free plan and see the current pricing for higher volumes.

FAQ

Q: Is jsoup enough on its own, or do I need Selenium?

For server-rendered HTML, jsoup alone is enough and considerably faster, since it never starts a browser. You need a rendering step only when the data is created by JavaScript — and as the measurement above shows, that step can be an HTTP call that returns rendered HTML rather than a full browser in your process.

Q: How do I tell whether a page needs JavaScript before writing selectors?

Print doc.html() from a plain jsoup fetch and search it for a value you can see on the page. If the value is absent from that string but visible in the browser, it is being rendered client-side. Comparing a selector count against the visible count catches the same thing.

Q: What is the practical reason to change maxBodySize?

The default keeps 2 MiB, which is smaller than many list, archive, and comparison pages. If your target exceeds it, everything past the cut is missing from the parsed tree with no error raised. Set maxBodySize(0) when you need the whole document, and compare received bytes against Content-Length to know when it matters.

Q: Does jsoup handle malformed HTML?

Yes. It applies the same error-recovery rules a browser does, so unclosed tags and misnested elements still produce a queryable tree. That is the main reason to prefer it over a strict XML parser for real-world pages.

Q: Is scraping with Java legal?

The language is irrelevant to the legal question. What matters is the data you collect, the terms of the site, the robots directives you respect, and the jurisdiction you operate in. Restrict collection to public pages, keep request volume modest, and take legal advice for anything involving personal data.