How to Extract IndexedDB Data With Playwright and Scrapeless

TL;DR:

• IndexedDB can contain structured browser data that never appears in the initial HTML. Reading it requires executing code inside the page’s origin after the application opens its database.
• IndexedDB extraction should begin with database and schema discovery. Enumerate databases, object stores, and indexes before reading records instead of assuming their names.
• Playwright can bridge IndexedDB’s callback-based requests into an awaitable extraction workflow. Wrap requests such as indexedDB.open() and getAll() in Promises inside page.evaluate().
• The rendered DOM and IndexedDB may expose different subsets of the same application state. Preserve both counts to detect filtering, pagination, or stale UI data.
• Large or sensitive stores require bounded extraction. Prefer key ranges, counts, or cursors over an unrestricted getAll() call, and inspect only authorized browser sessions.
• The same IndexedDB query works locally and through Scrapeless. Moving to Scrapeless Scraping Browser changes browser creation, not the page-side database logic.
• Free to start. New Scrapeless accounts include free Scraping Browser runtime—sign up at app.scrapeless.com.

Introduction: Browser Data Does Not End With the DOM

IndexedDB can hold structured records that never appear in a page's initial HTML. This guide uses Playwright to enumerate a real public database, inspect its object store and indexes, read all records, and compare them with the rendered table before moving the same extraction to the Scrapeless Scraping Browser.

What IndexedDB Adds to Web Extraction

IndexedDB is an asynchronous, origin-scoped database built into the browser. Unlike localStorage's string map, it stores structured JavaScript values in object stores and supports indexes, keys, and transactions. MDN's IndexedDB guide describes that model.

Applications use it for offline records, cached API data, queues, and state that can be too large or structured for Web Storage. A browser extractor can read public application state after the page has opened its database. That access does not make private user databases appropriate collection targets; this tutorial uses MDN's public contacts fixture only.

Why Use Playwright With Scrapeless

Playwright evaluates an asynchronous function in the page origin, so the function can call indexedDB.open, await request callbacks, and return plain data to Python. The Scrapeless Scraping Browser keeps that page-side API inside a hosted Chromium session.

Playwright connects through connect_over_cdp. Once the target page is loaded, the IndexedDB query is unchanged.

Prerequisites

• Python 3.10 or later.
• playwright 1.59.0 or a current compatible version.
• Local Chrome/Chromium for the complete runnable path.
• A funded Scrapeless account and SCRAPELESS_API_KEY for the cloud connection. The verification account returned code 14500 for new browser sessions, so that connection is a labelled prerequisite gap.

Install

python -m pip install "playwright==1.59.0"

Connect to the Scrapeless Scraping Browser

Note: This connection requires funded Scraping Browser balance. The final verification account returned insufficient balance, please recharge first. The complete IndexedDB extraction below ran in local Chrome.

import os
from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

params = urlencode({
    "token": os.environ["SCRAPELESS_API_KEY"],
    "sessionTTL": 120,
    "proxyCountry": "US",
})
cdp_url = f"wss://browser.scrapeless.com/api/v2/browser?{params}"

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(cdp_url)
    page = browser.contexts[0].pages[0]
    # Navigate and evaluate the IndexedDB query below.
    browser.close()

Use the Scrapeless developer docs for current connection parameters.

Step 1 — Discover the Origin’s Databases

MDN's public IDBIndex example creates one database after load:

TARGET_URL = (
    "https://mdn.github.io/dom-examples/"
    "indexeddb-examples/idbindex/"
)

page.goto(TARGET_URL, wait_until="domcontentloaded")
page.wait_for_selector("tbody tr")
databases = page.evaluate("indexedDB.databases()")
print("databases:", databases)

databases: [{'name': 'contactsList', 'version': 1}]

indexedDB.databases() is described by the IDBFactory databases reference. Check browser support before relying on it across older engines.

Step 2 — Inspect the Object Store and Indexes

Open the discovered database and return schema metadata:

schema = page.evaluate("""async () => {
    const opened = await new Promise((resolve, reject) => {
        const request = indexedDB.open('contactsList');
        request.onsuccess = () => resolve(request.result);
        request.onerror = () => reject(request.error);
    });
    const storeName = opened.objectStoreNames[0];
    const transaction = opened.transaction(storeName, 'readonly');
    const store = transaction.objectStore(storeName);
    const result = {
        storeName,
        indexes: Array.from(store.indexNames),
    };
    opened.close();
    return result;
}""")

print("object store:", schema["storeName"])
print("indexes:", schema["indexes"])

object store: contactsList
indexes: ['age', 'company', 'eMail', 'fName', 'jTitle', 'lName', 'phone']

This fixture uses contactsList for both the database and its single object store. Enumerating both still matters: many applications use different names, and assuming they match can target the wrong store.

Step 3 — Read Records From the Object Store

IDBObjectStore.getAll() returns a request, not a Promise. Wrap its success/error callbacks so Playwright can await it:

records = page.evaluate("""async () => {
    const database = await new Promise((resolve, reject) => {
        const request = indexedDB.open('contactsList');
        request.onsuccess = () => resolve(request.result);
        request.onerror = () => reject(request.error);
    });
    const transaction = database.transaction('contactsList', 'readonly');
    const store = transaction.objectStore('contactsList');
    const rows = await new Promise((resolve, reject) => {
        const request = store.getAll();
        request.onsuccess = () => resolve(request.result);
        request.onerror = () => reject(request.error);
    });
    database.close();
    return rows;
}""")

print("record count:", len(records))
print("first:", records[0])
print("last:", records[-1])

The getAll reference notes that it returns all records when no query or count is provided. Keep reads bounded on real applications; a cursor or count is safer for large stores.

Step 4 — Compare IndexedDB With the Rendered DOM

table_rows = page.locator("tbody tr").count()
arkham_rows = [row for row in records if row["company"] == "Arkham"]
ages = [row["age"] for row in records]

print("table rows:", table_rows)
print("database rows:", len(records))
print("Arkham rows:", len(arkham_rows))
print("age range:", min(ages), max(ages))

table rows: 10
database rows: 10
Arkham rows: 2
age range: 24 55

Matching counts prove that this fixture rendered the complete database. A different application may render only the current page or a filtered subset, so preserve both counts rather than forcing equality.

Ready to query browser state in a hosted session? Create a free Scrapeless account and replace only browser creation.

Complete Runnable Extractor

from playwright.sync_api import sync_playwright

TARGET_URL = (
    "https://mdn.github.io/dom-examples/"
    "indexeddb-examples/idbindex/"
)

QUERY = """async () => {
    const databases = await indexedDB.databases();
    const database = await new Promise((resolve, reject) => {
        const request = indexedDB.open('contactsList');
        request.onsuccess = () => resolve(request.result);
        request.onerror = () => reject(request.error);
    });
    const storeName = database.objectStoreNames[0];
    const transaction = database.transaction(storeName, 'readonly');
    const store = transaction.objectStore(storeName);
    const indexes = Array.from(store.indexNames);
    const rows = await new Promise((resolve, reject) => {
        const request = store.getAll();
        request.onsuccess = () => resolve(request.result);
        request.onerror = () => reject(request.error);
    });
    database.close();
    return {databases, storeName, indexes, rows};
}"""

with sync_playwright() as p:
    browser = p.chromium.launch(
        executable_path="/usr/bin/google-chrome",
        headless=True,
    )
    page = browser.new_page()
    page.goto(TARGET_URL, wait_until="domcontentloaded")
    page.wait_for_selector("tbody tr")

    result = page.evaluate(QUERY)
    rows = result["rows"]
    table_rows = page.locator("tbody tr").count()
    arkham_rows = [row for row in rows if row["company"] == "Arkham"]
    ages = [row["age"] for row in rows]

    assert result["databases"] == [{"name": "contactsList", "version": 1}]
    assert result["storeName"] == "contactsList"
    assert len(result["indexes"]) == 7
    assert len(rows) == table_rows == 10
    assert len(arkham_rows) == 2

    print("title:", page.title())
    print("databases:", result["databases"])
    print("object store:", result["storeName"])
    print("indexes:", result["indexes"])
    print("database rows:", len(rows))
    print("table rows:", table_rows)
    print("first contact:", rows[0]["fName"], rows[0]["lName"])
    print("last contact:", rows[-1]["fName"], rows[-1]["lName"])
    print("Arkham rows:", len(arkham_rows))
    print("age range:", min(ages), max(ages))
    browser.close()

The live result contains one version-1 database, one contactsList store, seven indexes, ten database records, ten rendered rows, two Arkham contacts, and ages from 24 to 55.

What You Get Back

The complete extractor returns the database identity, schema metadata, stored records, and a DOM comparison from the same browser session:

{
  "database": {
    "name": "contactsList",
    "version": 1
  },
  "object_store": "contactsList",
  "index_count": 7,
  "database_rows": 10,
  "rendered_rows": 10,
  "matching_company_rows": 2,
  "age_range": {
    "minimum": 24,
    "maximum": 55
  }
}

The matching database and table counts show that the public fixture rendered every stored record. Production applications may render only a filtered or paginated subset, so keep the database identity, object-store name, page URL, and both row counts with the extracted result.

Common IndexedDB Extraction Problems

The database list is empty

Wait for the application to open or create its database. An empty list immediately after navigation may indicate timing, an unsupported discovery API, or the wrong origin.

The object store is large

Do not call getAll() without a limit on an unbounded store. Use a key range, count, or cursor and stop after the dataset slice your task needs.

Values contain non-JSON types

Playwright serializes supported values back to Python. Blobs, complex class instances, and cyclic structures may need field-level mapping inside the evaluated function.

The page and database disagree

The page may be filtered, paginated, or stale. Store the database identity, object-store name, page URL, and extraction time so the mismatch can be investigated.

Safe Extraction Boundaries

IndexedDB frequently contains private offline application data. Use this technique only on systems and sessions you are authorized to inspect. Do not publish credentials, messages, or personal records. The public MDN contacts are synthetic fixture data.

Conclusion

IndexedDB extraction starts with discovery and schema inspection, not a guessed store name. Open the origin's database, enumerate stores and indexes, bridge request callbacks into a Promise, and validate the result against a visible surface where one exists. Moving the workflow to Scrapeless changes browser creation while preserving the page-side query.

Ready to Extract Structured Browser State?

Join developers building browser-based extraction workflows in the Scrapeless community: Discord · Telegram.

Start with Scrapeless, review Scrapeless pricing, and read what CDP exposes before moving the same IndexedDB workflow to a hosted browser.

FAQ

Q: Can Playwright read IndexedDB?

Yes. Evaluate an asynchronous function in the page origin and wrap IndexedDB request callbacks in Promises.

Q: Why enumerate databases first?

It confirms the origin and database identity before you assume a name or version. Not every browser exposes indexedDB.databases(), so keep a known-name fallback when compatibility requires it.

Q: Is an object store the same as a database?

No. A database contains one or more object stores. In the MDN fixture, both the database and its object store are named contactsList.

Q: When should I use a cursor instead of getAll?

Use a cursor or bounded query when a store may be large, when you need ordering, or when you can stop after a small subset.

Q: Why compare IndexedDB rows with the DOM?

The comparison reveals whether the UI shows every record, a filtered view, or only one page. Neither surface should silently overwrite the other.

Q: Does IndexedDB persist across browser sessions?

It can persist in a retained browser profile until the origin or browser clears it. An ephemeral context may disappear when the context closes.

Q: Is extracting IndexedDB data always permitted?

No. Inspect only authorized sessions and necessary public data, minimize retained fields, follow the site's terms and robots policy where applicable, and obtain legal advice for sensitive uses.