niquests Web Scraping: Modern HTTP/2 Requests for Python

TL;DR:

• niquests is a drop-in replacement for requests with the same API plus HTTP/2, HTTP/3, and connection multiplexing — change the import and your code keeps working.
• What niquests does not do is render JavaScript. It is an HTTP client; on a script-built page it returns the markup the server sent, which holds none of the content.
• The gap shows up in one script. niquests fetches a JavaScript-rendered demo page over HTTP/2 and finds 0 quote blocks in it.
• niquests is also the client that calls Scrapeless. A live run used the same session to POST the URL to the Scrapeless Universal Scraping API, got the rendered HTML back, and pulled all 10 authors from it.
• One HTTP client for both jobs. Direct fetches where they work, the Scrapeless API where rendering is needed — same Session, same API.
• Free to start on the fetch side. Create your Scrapeless API key at app.scrapeless.com.

What niquests is, and what it is not

niquests is a fork of requests that keeps the API you already know — Session, get, post, json() — and adds the transport features requests never got: HTTP/2 and HTTP/3, connection multiplexing, and DNS-over-HTTPS. For a scraper, the practical win is that import niquests as requests upgrades an existing codebase to modern HTTP without a rewrite, and the multiplexed transport moves many requests over fewer connections.

What it is not is a browser. niquests speaks HTTP, so it returns exactly what the server sends; it does not run the JavaScript that builds a modern page's content. A faster, more modern transport does not change that — an empty page fetched over HTTP/2 is still empty. So a niquests scraper is two jobs for one client: fetch the pages that serve their content directly, and for the pages that build it with scripts, call a rendering API. This guide uses the Scrapeless Universal Scraping API for the second job, with niquests itself making that call. For the wider toolkit, the Python web scraping tutorial is the companion.

Install

niquests is the whole toolchain — it needs no separate parser for this guide. The version this guide was written against is niquests 3.20.1:

pip install "niquests==3.20.1"

Keep your key in the environment, never in source:

export SCRAPELESS_API_KEY="sk_your_scrapeless_key"

The direct fetch, and where it stops

Point niquests at a JavaScript-rendered page and two things are true at once: the transport is modern, and the content is missing. The connection negotiates HTTP/2 — a protocol defined by the HTTP/2 standard that requests does not speak — yet the returned markup has zero quote blocks, because the content is built client-side per the HTML scripting specification:

# direct.py — modern transport, missing content
import niquests

session = niquests.Session()
resp = session.get("https://quotes.toscrape.com/js/", timeout=30)

print("http version:", resp.conn_info.http_version)
print("js-page quote blocks:", resp.text.count('class="quote"'))

The transport is HTTP/2; the content is not there:

http version: HttpVersion.h2
js-page quote blocks: 0

That zero is not a niquests failure — it is the correct result for an HTTP client on a page whose content arrives after the scripts run. The fix is a fetch layer that renders.

The rendered fetch, with niquests calling Scrapeless

Keep the same session and change the target: instead of the page, POST its URL to the Scrapeless Universal Scraping API, which renders server-side and returns the finished HTML. niquests handles this request like any other, so one client covers both paths — the request and response layer here follows the HTTP semantics standard:

# rendered.py — niquests calls the render API, then extracts
import os
import re

import niquests

session = niquests.Session()
resp = session.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker", "input": {"url": "https://quotes.toscrape.com/js/", "js_render": True}},
    timeout=120,
)
resp.raise_for_status()
html = resp.json()["data"]

authors = re.findall(r'<small class="author">(.*?)</small>', html)
print("rendered quote blocks:", html.count('class="quote"'))
print("authors extracted:", len(authors))
print("first author:", authors[0])

Now the content is present and extractable:

rendered quote blocks: 10
authors extracted: 10
first author: Albert Einstein

That is the whole scraper, and it never left niquests: one Session making a direct request where that works and a Scrapeless request where rendering is needed. The Universal Scraping API does the rendering; niquests does the HTTP.

Get your API key on the free plan: app.scrapeless.com

Advanced patterns

• Migrate with one line. import niquests as requests lets an existing requests codebase adopt HTTP/2 and multiplexing with no other change; the API is the same.
• Reuse the session. A single niquests.Session() pools and multiplexes connections, which is where the transport advantage shows up across many requests — build it once and pass it around.
• Add a real parser when you outgrow regex. The regex here keeps the example to one dependency; for anything beyond a flat list, feed resp.json()["data"] to a selector library and select structured fields.
• Let it negotiate. niquests picks HTTP/2 or HTTP/3 when the server offers it and falls back cleanly when it does not; you do not configure the version, you read it off conn_info when you want to confirm.

Troubleshooting

• conn_info.http_version is HTTP/1.1. The server did not offer HTTP/2 for that request, or an intermediary downgraded it. This is normal and not an error; niquests uses the best version available.
• Zero blocks from a page you can see in a browser. The content is JavaScript-rendered and the direct fetch returned pre-render HTML — the js-page quote blocks: 0 case. Route that URL through the Scrapeless call with js_render instead.
• json() raises on the Scrapeless response. The request failed before rendering. Call raise_for_status() first, and confirm the key is set and the actor and input are shaped as shown.
• Timeouts on slow pages. Rendering takes longer than a static fetch. Give the Scrapeless POST a generous timeout, as the example does, rather than the short default you would use for a direct request.

Conclusion

niquests earns its place as the one HTTP client a scraper needs: the requests API with modern transport, handling both the direct fetch and the call to a rendering API. The layer that turns a script-built page into content is the fetch — the direct request's zero-blocks result settles that — and one Scrapeless POST, made by niquests itself, closes the gap. Wire the two paths into a single session and the demo page's ten authors come back over HTTP you did not have to rewrite.

Create a free Scrapeless account to get an API key, and the developer docs cover the unlocker.webunlocker parameters. Check Scrapeless pricing when you plan a recurring job.

FAQ

Q: Can niquests scrape JavaScript-rendered pages by itself?

No. niquests is an HTTP client, not a browser; it returns the markup the server sends and runs no scripts. On a page that builds its content client-side, the direct fetch comes back with the content missing — the guide's zero-blocks result. Route those pages through the Scrapeless Universal Scraping API, which renders them, and niquests makes that call too.

Q: How is niquests different from requests?

Same API, newer transport. niquests is a requests fork that adds HTTP/2, HTTP/3, connection multiplexing, and DNS-over-HTTPS while keeping Session, get, post, and json() identical. import niquests as requests is usually the entire migration.

Q: Do I need a separate parser?

For a flat list of fields, a regex or the standard library is enough, as shown. For nested or structured extraction, pair niquests with a selector library — niquests fetches, the parser selects. This guide keeps to one dependency on purpose.

Q: Does HTTP/2 help with scraping blocks?

It is a transport upgrade, not an anti-blocking measure. HTTP/2 multiplexing improves throughput across many requests, but it does not render JavaScript or clear access challenges — that is the fetch layer's job, which is why the rendered path goes through Scrapeless.

Q: Is scraping with niquests legal?

The HTTP client does not change the collection rules. Fetch public pages only, respect site terms and the robots directives standardized by the Robots Exclusion Protocol, keep volumes bounded, and handle any personal data under the laws that apply to you.