How to Crawl Infinite-Scroll Product Collections with Playwright

Scrapeless Agent Browser runs Playwright-driven cloud sessions, and Scrapeless Residential Proxies provide location-targeted routes for eligible collection jobs.

Identify What Loads the Next Collection Page

An infinite-scroll product collection asks for more items when the user reaches a trigger, which may be the page edge, a sentinel element or the bottom of a nested container. The Intersection Observer specification describes how a page can observe a target crossing a viewport or container boundary. DevTools can help you identify which element actually scrolls and whether a fetch follows that action. Check after a filter change too. Search, sorting and stock filters can each start a separate collection.

Look for a permitted structured source first. A store-owned feed or supported pagination endpoint may expose a cursor, stable item identifier and end flag directly. Preserve any filters that define the collection. When the user experience itself is the target, use the browser to see how the next batch loads and what state tells the customer that the collection ended. Treat data obtained from each surface according to the scope agreed with the owner.

The apparent page height can be misleading. A virtualized list maintains a small set of mounted cards and recycles them as the user scrolls. Its height may be reserved from the beginning. In that case, the signal of progress is a newly observed item ID or batch cursor, and the signal of completion comes from the application. The position of the last visible row is only one clue.

Prerequisites and Installation

Use a Node.js environment with Playwright and Chromium. Add the Playwright package to a small project, install its Chromium browser or use a configured Chromium executable, and start with a storefront fixture you control. In a real integration, inspect the markup for durable card IDs and a completion signal. The example's test IDs are fixture attributes. They are expected to change when you adapt the function to an actual store.

Use Playwright locators to scope product elements to the collection. A broad link selector can pick up navigation, editorial cards and recommendation carousels. Prefer an immutable product ID on the card or a verified URL pattern. Preserve the original link for evidence and normalize URL components only under known rules. The URL Standard describes the browser's URL parsing and serialization behavior.

Collect Each Batch Before It Leaves the DOM

Read the current cards into a map keyed by stable ID before moving the scroll position. Store the displayed title and the resolved product URL with the ID. When a card reappears because the user scrolls back, the map retains one discovery row for that product. If multiple variants intentionally appear as separate cards, use a key that represents the desired discovery unit.

The following example runs against a controlled storefront fixture with a scrollable panel, a batch marker, two product cards mounted in sequence and an explicit end marker. Use the same pattern with store-specific selectors after confirming permissions and pagination behavior. A live Agent Browser session requires a Scrapeless account and the connection setup described in the quickstart.

async function collect(page, targetUrl, maxRounds = 12) {
  await page.goto(targetUrl, { waitUntil: 'domcontentloaded' });
  const grid = page.getByTestId('collection');
  const cards = grid.locator('[data-product-id]');
  const scroller = page.getByTestId('scroll-pane');
  const seen = new Map();

  for (let round = 0; round < maxRounds; round += 1) {
    await cards.first().waitFor({ state: 'visible' });
    const batch = await cards.evaluateAll(nodes => nodes.map(node => ({
      id: node.getAttribute('data-product-id'),
      title: node.querySelector('a')?.textContent.trim(),
      url: node.querySelector('a')?.href
    })));
    for (const item of batch) {
      if (item.id && item.title && item.url) seen.set(item.id, item);
    }
    if (await page.getByTestId('collection-end').isVisible()) {
      return { status: 'complete', items: [...seen.values()] };
    }
    const batchId = await grid.getAttribute('data-batch');
    if (!batchId) throw new Error('Collection batch identifier is missing');
    await scroller.hover();
    await page.mouse.wheel(0, 240);
    try {
      await page.waitForFunction(previous => {
        const root = document.querySelector('[data-testid="collection"]');
        return root?.getAttribute('data-batch') !== previous;
      }, batchId, { timeout: 3000 });
    } catch (error) {
      if (error.name !== 'TimeoutError') throw error;
      return { status: 'no_progress', items: [...seen.values()] };
    }
  }
  return { status: 'round_limit', items: [...seen.values()] };
}

The fixture returns item IDs A and B with a complete status after the first card is removed from the DOM. The example's batch marker is a fixture implementation detail. For a store you operate, a successful request tied to the same filter and cursor can serve as a useful progress signal once the corresponding cards render. A delayed change should result in a bounded no-progress outcome. That outcome leaves the partial list available for inspection.

Advance the Right Scroll Container

The next action must target the element responsible for loading more cards. A collection embedded in a modal or category pane may ignore window scrolling. In the example, hovering the pane puts the wheel input over the correct region. Use a scroll increment that preserves overlap across mounted rows and measure whether you miss IDs between steps. Virtualized grids can skip intermediate cards when a jump is too large.

Observe loading indicators and distinguish a loaded empty collection from a render that has never produced cards. An intentional empty state can be a complete result with zero products. A skeleton, access challenge, consent dialog or broken selector should generate an explanatory outcome. Extend the fixture to cover those paths before treating empty results as healthy.

A load-more button needs a different action but uses the same progress model. Select the button within the collection, click once, then wait for the new batch or end state. Avoid global inactivity as the completion condition. Long-running requests outside the grid can remain active while the visible collection is already ready.

Track Completeness Across Filters and Store Changes

A crawl is complete for its collection configuration and its observed window. Store the requested category, market, sort, filter parameters, final URL, discovered count, completion reason and collector version. If an advertised result count exists, record it as a separate comparison. A storefront may change while the job runs, so a mismatch can reflect catalog churn or duplicate display as well as a collector defect.

Keep discovery independent from offer extraction. A collection card may represent a product family and display a starting price. Pass stable discovery IDs and links into a separate detail-page worker that resolves seller, variant, price and stock. The infinite-scroll scraping walkthrough covers more pagination patterns, and the price monitoring solution for regional comparisons is relevant when the discovered product set depends on market.

Test pagination with a small collection that has known expected identities. Add a fixture where the number of mounted nodes remains constant across batches. Then exercise a missing end marker and a stalled batch. Compare the returned status and item IDs against the fixture data. Add filter-specific tests if the same adapter covers a category and its search results.

Move Browser Execution to the Cloud Where Needed

A Playwright collector can run on a local browser or in Scrapeless Agent Browser. The Agent Browser Playwright quickstart shows the supported connection setup for remote sessions. Use the Agent Browser solution for dynamic websites to plan capacity when each collection needs rendered controls and a sequence of input events. Keep the crawl state and its last accepted batch in your application.

Use Residential Proxies where geography is needed for an authorized observation, and verify the market that the storefront actually renders. A pinned region does not specify the collection's sort or filter. Store those settings independently and keep each market run isolated when cookies can affect the catalog. Estimate cloud browser time and proxy traffic against Scrapeless pricing after measuring completed and partial jobs in the same sample.

Report What the Crawl Actually Saw

Use explicit progress markers and retain a stable item map. Put the completion reason beside the discovered URLs so a downstream job can distinguish a full collection from a partial result. Start with one known category and add sources only after the fixture covers virtualization, an empty collection and a stalled loader.