Scraping Infinite Scroll: Stop Guessing How Many Times to Scroll

TL;DR:

• An infinite-scroll page ships almost nothing in its HTML. The demo used here serves 2,671 bytes containing zero quote elements; every item arrives afterwards.
• A fixed scroll count is the failure everyone ships. Five scrolls returned 60 of 100 items — no error, no warning, a clean exit with 40% of the data missing.
• Scroll until the item count stops growing instead. That returned all 100.
• The number of scrolls is not a property of the site. The same loop needed 11 scroll actions locally and 3 on a cloud browser, reaching 100 both times.
• The page made 10 calls to /api/quotes?page=N. Reading that endpoint directly returns the same 100 items and reports has_next, so the loop knows when it is finished.
• The Scrapeless free plan covers the cloud-browser run in this guide.

Infinite scroll breaks scrapers quietly. The request succeeds, the selectors match, the script exits zero — and the dataset is short. Nothing in the run tells you how much you missed, because the page never promised how much there was.

This guide measures that on a live demo page. It builds the loop most tutorials ship, shows exactly what it drops, replaces it with one that has a real stopping condition, and then finds the request the page was making all along.

What the Server Actually Sends

Fetch the page without a browser and there is nothing to parse:

def served_html() -> tuple[int, int]:
    request = urllib.request.Request(SCROLL_URL, headers={"User-Agent": UA})
    with urllib.request.urlopen(request, timeout=45) as response:
        html = response.read().decode("utf-8", "replace")
    return len(html), html.count('class="quote"')

scroll page bytes        : 2671
quote elements in the html: 0

2,671 bytes and not one item. The markup is a shell; the content is fetched by script after the page loads, typically when a sentinel element near the bottom enters view — the mechanism the Intersection Observer specification defines. A CSS selector cannot fail more cleanly than this — it matches nothing because nothing is there yet.

Install

pip install playwright
playwright install chromium

The second command downloads the browser binary; the pip package alone will not launch. The verification run used Playwright 1.59.0.

The Loop Everyone Ships

The standard recipe is to scroll a fixed number of times and then read the page. page.mouse.wheel() dispatches a real wheel event of the kind the UI Events specification defines, which is what the page's own listener is waiting for:

def scroll_fixed(page, times: int, pause: float) -> int:
    for _ in range(times):
        page.mouse.wheel(0, 20000)
        time.sleep(pause)
    return page.locator("div.quote").count()

Five scrolls against the live page:

quotes after 5 scrolls   : 60
elapsed seconds          : 5.1

Sixty items. The page holds a hundred. The script raised nothing, the selector worked, and the run looks successful from the outside — which is what makes this the expensive version of the bug. Pick five because it worked once, and every later run inherits a silent 40% shortfall.

Raising the number is not a fix either. It trades under-collection for wasted scrolls, and it still encodes a guess about a site that can change its batch size whenever it likes.

Scroll Until It Stops Growing

The condition you actually want is observable: keep scrolling while the item count is still increasing, and stop once it holds steady.

def scroll_until_stable(page, pause: float, no_growth_limit: int = 2) -> tuple[int, int]:
    seen = page.locator("div.quote").count()
    scrolls = 0
    stable = 0
    while stable < no_growth_limit:
        page.mouse.wheel(0, 20000)
        time.sleep(pause)
        scrolls += 1
        count = page.locator("div.quote").count()
        if count == seen:
            stable += 1
        else:
            stable = 0
            seen = count
    return seen, scrolls

no_growth_limit decides how much evidence you need before believing the page is finished. One flat reading is not proof — a batch still in flight looks identical to the end of the list. Requiring two consecutive flat readings costs one extra scroll and removes that ambiguity.

quotes collected         : 100
scroll actions performed : 11
elapsed seconds          : 11.1
api calls the page made  : 10
first api url            : https://quotes.toscrape.com/api/quotes?page=1

All 100, and the loop discovered the count rather than assuming it. Note the elapsed figure is mostly the pause this loop chooses to wait — eleven scrolls at one second each. It is a property of the loop, not of the site.

The Scroll Count Is Not a Property of the Site

Running the identical function against a cloud browser instead of local Chromium:

quotes collected         : 100
scroll actions performed : 3

Same function, same page, same 100 items — 3 scroll actions instead of 11. Viewport height and how quickly each batch arrives decide how much one wheel event advances the page, and neither is under your control. The viewport dimensions that scrolling is measured against are the ones the CSSOM View Module specifies. Any hard-coded scroll count is calibrated to one machine's window size.

That run used the Scrapeless Scraping Browser, which Playwright connects to over the Chrome DevTools Protocol. Only the connection line changes:

    endpoint = (
        "wss://browser.scrapeless.com/api/v2/browser"
        f"?token={os.environ['SCRAPELESS_API_KEY']}&sessionTTL=180&proxyCountry=ANY"
    )
    with sync_playwright() as p:
        browser = p.chromium.connect_over_cdp(endpoint, timeout=90000)
        page = browser.new_page()
        page.goto(SCROLL_URL, wait_until="domcontentloaded")
        cloud_total, cloud_scrolls = scroll_until_stable(page, pause=1.0)

connect_over_cdp replaces chromium.launch() and speaks the Chrome DevTools Protocol over a socket instead of to a local process, so the scroll loop, selectors, and extraction stay exactly as they were. Keep the key in the environment as SCRAPELESS_API_KEY; the Scraping Browser introduction documents the remaining session parameters.

Getting started takes a minute — create a free Scrapeless account and the free plan covers this run.

Watch What the Page Is Requesting

The scroll loop is a way of asking the page to fetch data. Listening to the requests shows what it fetches:

        calls: list[str] = []
        context = browser.new_context(user_agent=UA)
        context.on("request", lambda r: calls.append(r.url) if "/api/quotes" in r.url else None)

Ten calls, the first being https://quotes.toscrape.com/api/quotes?page=1. The page is paginating a JSON endpoint and rendering the results; scrolling is just the trigger.

That endpoint can be read directly, and it answers the question the scroll loop had to infer:

def read_api() -> tuple[int, int]:
    quotes = 0
    page = 1
    while True:
        request = urllib.request.Request(API_URL.format(page=page), headers={"User-Agent": UA})
        with urllib.request.urlopen(request, timeout=45) as response:
            payload = json.loads(response.read().decode())
        quotes += len(payload["quotes"])
        if not payload["has_next"]:
            return quotes, page
        page += 1

quotes collected         : 100
api pages requested      : 10
elapsed seconds          : 3.8

The same 100 items, with has_next as an explicit termination condition rather than "the count stopped changing." The response is already structured, so no selector sits between you and the fields — which also means no selector to break when the markup is restyled.

This is worth checking before writing any scroll loop. It does not always exist: plenty of sites render server-side on scroll, sign their requests, or return HTML fragments rather than JSON. When it does exist, it is the more durable target, and the browser is still what shows you it is there. For the other pagination shapes — next buttons, numbered pages, load-more — the full pagination guide covers each type.

Run It

export SCRAPELESS_API_KEY="your-api-key"
python3 scroll_demo.py

The complete output from the verification run:

playwright 1.59.0
--- what the server actually sends ---
scroll page bytes        : 2671
quote elements in the html: 0
--- fixed number of scrolls ---
quotes after 5 scrolls   : 60
elapsed seconds          : 5.1
--- scroll until the count stops growing ---
quotes collected         : 100
scroll actions performed : 11
elapsed seconds          : 11.1
api calls the page made  : 10
first api url            : https://quotes.toscrape.com/api/quotes?page=1
--- the same loop on the Scraping Browser ---
quotes collected         : 100
scroll actions performed : 3
--- reading the same api directly ---
quotes collected         : 100
api pages requested      : 10
elapsed seconds          : 3.8
browser/api time ratio   : 3x

The ratio on the last line compares the browser loop's wall clock against the direct reads. Most of the browser side is the deliberate one-second pause between scrolls, so read it as the cost of polling a page for a condition it never reports — not as a benchmark of the browser itself.

Troubleshooting

The count never stops growing. Some pages loop their content. Cap the loop with a maximum scroll count alongside the stability check, and treat hitting that cap as a signal to inspect the page rather than as a completed run.

Zero items even after scrolling. The container may scroll instead of the window. Scroll the element itself with page.locator(...).hover() followed by page.mouse.wheel(...), or call scroll_into_view_if_needed() on the last item.

The count grows, then the page goes blank. Long lists are often virtualized, so rows are removed from the DOM as they leave the viewport. Collect items as you go instead of reading them all at the end.

It works headed and not headless. Viewport size differs between the two, which changes how far one wheel event travels and whether the loader enters view. Set an explicit viewport on the context.

playwright._impl._errors.Error: Executable doesn't exist. The browser binary was never downloaded. Run playwright install chromium.

Conclusion

Infinite scroll turns "did I get everything?" into a question your scraper has to answer for itself, and a fixed scroll count answers it by guessing. The measurements here show what that costs: 60 items out of 100 from five scrolls, and a scroll count that changed from 11 to 3 purely by moving to a different browser.

Loop on an observed condition — item count stable across consecutive reads — rather than a number you chose. And before writing the loop at all, watch the page's requests: when the content arrives as JSON with a has_next flag, the page has already told you how to know when you are done.

Ready to try it? Start with the Scrapeless free plan and see the current pricing for higher volumes.

FAQ

Q: How many times should I scroll?

Do not pick a number. The measurements above used one loop against two browsers and needed 11 scrolls in one and 3 in the other for the same 100 items, because viewport height and batch timing decide how far each wheel event goes. Loop while the item count is still growing and stop after it holds steady twice.

Q: How do I know the page has finished loading everything?

Two consecutive reads with an unchanged item count is the practical signal, because a single flat reading is indistinguishable from a batch still in flight. If the page's underlying request exposes a flag such as has_next, that is a definite answer rather than an inference.

Q: Do I need a browser for infinite scroll at all?

Often not. The demo page here loads its content from a JSON endpoint that can be read directly for all 100 items. The browser is still how you discover that endpoint — attach a request listener, scroll once, and read the URLs. Sites that render server-side on scroll or sign their requests do need the browser.

Q: Why does my scraper return fewer items than the page displays?

Either the loop stopped early, or the list is virtualized and rows were removed from the DOM as they scrolled out of view. Print the item count after every scroll: a count that climbs and then falls indicates virtualization, and a count still climbing when the loop ends indicates the loop stopped too soon.

Q: Does wait_until="networkidle" solve this?

No. It waits for the initial load to settle, which happens before any scrolling has triggered a fetch. The batches arrive in response to scroll events, so the page can be idle and nearly empty at the same time — which is exactly the 2,671-byte state measured at the start.