How to Build a WooCommerce Product Scraper with Playwright

Scrapeless Agent Browser runs Playwright automation for WooCommerce storefronts, and Scrapeless Residential Proxies support market-specific network routing where store prices vary by location.

Choose the WooCommerce Surface That Answers the Question

A WooCommerce product scraper uses catalog sources to discover products and Playwright to confirm an interactive offer when the rendered storefront matters. On a store you own or are authorized to collect, check the WooCommerce Store API Products resource for public catalog fields and filters. The Products resource documents how published and password-protected products are represented and how product variation records differ from the default product collection.

Use an approved catalog feed if it supplies all the required fields. A browser is appropriate when a theme or extension changes the displayed price, stock message or variant choice after interaction. Keep the provenance of each field clear. A store API amount and the visible product price may serve different purposes if one is a base value and the other has a store-defined display adjustment.

The approach works for catalog QA, monitoring selected public offers, checking that variable products show the intended information, and collecting screenshots for merchandising review. Define which output the team expects before scraping. A fixture suitable for testing price selection can omit marketing fields and still be representative of the variation workflow.

Prerequisites and Installation

Use Node.js with Playwright and a Chromium runtime. Install the Playwright package and its Chromium binary or use an available compatible Chromium executable. You need a store you operate or have permission to collect and a sample with simple products, variable products and unavailable combinations. The example below uses a controlled fixture with stable attributes, so it can be run before adapting selectors to a WooCommerce theme.

For remote execution, use the Agent Browser Playwright quickstart and configure a Scrapeless account and API key as documented. Keep the credential out of target URLs and logs. The locally tested example takes an existing Playwright page. A remote run also needs an authorized target reachable from the cloud session and any agreed location or market settings.

Step 1. Discover Products and Preserve Source IDs

Discover each target from an approved sitemap, storefront navigation, store feed or documented catalog source. Keep the store namespace and product ID where available. Save the canonical route for deduplication and the original route used in the visit. The URL Standard defines how relative links resolve. Preserve variation query parameters or route components when they identify the storefront state you plan to verify.

Treat catalog pagination separately from product extraction. Record the filters and cursor or page for each discovery run. Use durable IDs to deduplicate results, and set a clear upper limit and completion rule. This prevents listing pages from being confused with product detail pages and gives the team an audit trail if one category appears to lose products.

Step 2. Identify How the Theme Confirms a Variation

A variation capture starts by identifying the option controls and the signal that commits their final state. Inspect the label and input behavior. A theme may use native selects, swatches or custom buttons. When the collector selects a combination, watch for a variation identifier, resolved stock state or visible amount tied to that combination. Scope extraction to the product detail region so a sale banner or related item does not supply the amount.

The following Playwright function reads a controlled WooCommerce-style product fixture. The test IDs and readiness attributes are part of that fixture. On a real WooCommerce site, adapt the selectors to a known theme or ask the store team to expose a stable readiness marker. The example exercises an asynchronous price change for a selected blue mug.

async function readWooProduct(page, targetUrl, selections) {
  const response = await page.goto(targetUrl, { waitUntil: 'domcontentloaded' });
  if (!response?.ok()) throw new Error('Product navigation failed');

  const product = page.getByTestId('product-detail');
  const title = await product.getByRole('heading', { level: 1 }).innerText();
  for (const [label, value] of Object.entries(selections)) {
    await product.getByRole('combobox', { name: label }).selectOption(value);
  }

  const expected = JSON.stringify(selections);
  await page.waitForFunction(options => {
    const node = document.querySelector('[data-testid="resolved-offer"]');
    if (!node || node.dataset.state !== 'ready') return false;
    const selected = JSON.parse(node.dataset.options || '{}');
    return Object.entries(JSON.parse(options))
      .every(([name, value]) => selected[name] === value);
  }, expected, { timeout: 5000 });

  return product.getByTestId('resolved-offer').evaluate((node, heading) => ({
    title: heading,
    url: location.href,
    variantId: node.dataset.variantId,
    options: JSON.parse(node.dataset.options),
    priceText: node.querySelector('[data-testid="price"]')?.textContent.trim() || null,
    currency: node.dataset.currency || null,
    availability: node.dataset.availability || 'unknown'
  }), title);
}

The controlled fixture starts with the white option and resolves blue to variation 11 with a displayed amount of $21.00. The local test verifies the variation and amount together. A simple product can use an empty selection and a ready offer with its own product identity if the adapter contract defines that shape. When a variation is unavailable, capture its explicit stock message and allow the amount to be absent if the contract permits it.

Step 3. Interpret the Price With Its Context

A price field should distinguish the visible selling price from a crossed-out list price and a range. Keep the raw amount and currency before normalization. Use the Offer vocabulary for price and availability when naming the normalized fields. If taxes, fees, shipping or promotions matter, document how the current surface includes them. Avoid silently mixing them across markets.

An unavailable variant is still a useful catalog observation. Store its explicit state and selected combination. Give unresolved variant state a separate outcome so an empty price is never interpreted as zero. An empty selector might be caused by the active theme, an unsupported combination or a failure to reach the page. Check the surrounding evidence before assigning a stock label.

Use a single approved source of truth for amount normalization. If you use the Store API, interpret its amounts with the accompanying currency information. If you use the display, parse it according to the site's known locale and amount type. Compare the two for a small sample and record deliberate differences. This can catch store plugins that change a display field without updating the catalog feed.

Step 4. Run the Browser in the Right Market

Use Scrapeless Agent Browser when the job needs a managed browser session, and the managed web automation solution when planning interactive workflows across a portfolio of storefronts. The Playwright proxy setup guide explains how routing fits into browser configuration. For an authorized location-specific capture, select Residential Proxies and check currency, locale and delivery context after navigating.

A market set in a store selector can be stored in cookies or the URL. Start each independent market test with the intended session policy. Verify that the rendered market agrees before accepting the amount. Keep actual browser duration and transfer by template so you can compare against Scrapeless pricing. The price monitoring proxy solution is useful when the collection needs consistent region-by-region observations.

What the Collector Returns

A useful WooCommerce record contains the store, product or variation identifier, selected attributes, final URL, raw display price, currency, explicit availability and observation time. Record the source, the collection version and whether you used a simple or variable product path. Save incomplete outputs with a clear reason and separate them from validated offer events. This gives pricing or QA consumers a reliable way to distinguish a complete observation from a collector that reached the URL but could not verify the page.

Build the Adapter Around One Known Product Family

Start with a store you can inspect and a variation whose visible price you know. Exercise the product list, simple product and variable product paths separately, then compare the stored rows with the rendered storefront. Add markets and source templates after you can explain each field and outcome.