JSON-LD Web Scraping With Python: Extract Structured Data

TL;DR:

• JSON-LD is usually the shortest path from a page to a typed record. Look for <script type="application/ld+json"> before writing selectors for a headline, author, publication date, image, or product fields.
• Read every JSON-LD block. A page may split organization, breadcrumb, article, product, and FAQ data across several scripts.
• Normalize three top-level shapes. A JSON-LD block can be one object, an array of objects, or an object whose @graph contains the useful nodes.
• Schema.org fields are optional in practice. Select the node by @type, keep absent fields as None, and validate only the fields your pipeline truly requires.
• Beautiful Soup does not execute JavaScript. If a page injects JSON-LD after load, fetch rendered HTML first and run the same parser on that response.
• You can test the parser without a cloud model. The complete Python example below reads one real Article node, compares its headline with the visible H1, and validates the output.
• Start with public, bounded pages. Create a free Scrapeless account when your target needs rendered HTML.

JSON-LD often contains the fields a scraper is about to reconstruct from scattered page elements. On one live Scrapeless article, a single Article object carries the headline, author, publisher, dates, canonical URL, hero image, description, and keywords. The visible page still matters, but structured metadata gives the extraction pipeline a typed starting point.

JSON-LD follows the JSON-LD 1.1 specification, while vocabularies such as Article, Product, and BreadcrumbList come from Schema.org's structured-data model. Neither standard guarantees that every publisher fills every property. Your parser has to preserve that uncertainty instead of inventing values.

What JSON-LD Web Scraping Is Good For

JSON-LD is useful when a page publishes machine-readable facts alongside its human-facing layout. Common nodes include:

• Article and NewsArticle for headlines, dates, authors, images, and publishers;
• Product for names, brands, offers, ratings, and identifiers;
• BreadcrumbList for hierarchy and canonical category paths;
• Organization, Person, and LocalBusiness for entity metadata;
• VideoObject, Recipe, Event, and other domain-specific types.

The script block is not a private endpoint. It is part of the page response and is intended for machines such as search crawlers. That makes it more durable than a generated CSS class, but not automatically complete or correct. Treat it as a source to validate, not an oracle.

Install Beautiful Soup

This guide uses Python 3.10 or later, requests, and beautifulsoup4 4.15.0:

python -m pip install "requests>=2.32,<3" "beautifulsoup4==4.15.0"

Beautiful Soup's tree-search API can filter tags by attribute, which is enough to collect every matching script. JSON decoding stays in Python's standard library.

Fetch the Source HTML

Start with an ordinary HTTP request. The target used here publishes its JSON-LD in the initial response, so JavaScript rendering would add cost without changing the result:

import requests

URL = "https://www.scrapeless.com/en/blog/what-is-web-scraping?utm_source=website&utm_medium=blog&utm_campaign=universalscrapingapi&utm_term=json-ld-structured-data-web-scraping"

response = requests.get(
    URL,
    headers={"User-Agent": "Mozilla/5.0"},
    timeout=30,
)
response.raise_for_status()
print("HTML bytes:", len(response.content))

HTML bytes: 439487

The byte count can change when the page bundle changes. The useful invariant is that the response contains at least one decodable application/ld+json script.

Parse Every JSON-LD Block

Do not use soup.find(...) unless the page contract explicitly promises one block. find_all preserves the possibility that the article, breadcrumb, and publisher live in separate scripts:

import json
from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, "html.parser")
scripts = soup.find_all("script", type="application/ld+json")

parsed_blocks = []
for script in scripts:
    raw = script.get_text(strip=True)
    try:
        parsed_blocks.append(json.loads(raw))
    except json.JSONDecodeError:
        continue

print("JSON-LD blocks:", len(scripts))
print("decoded blocks:", len(parsed_blocks))

Skipping malformed blocks is safe only when you also record how many were skipped. A silent except can turn a metadata regression into an empty dataset that looks successful.

Normalize Objects, Arrays, and @graph

JSON-LD authors have several valid ways to group nodes. This generator flattens the three shapes a scraper meets most often:

def iter_nodes(value):
    if isinstance(value, list):
        for item in value:
            yield from iter_nodes(item)
    elif isinstance(value, dict):
        graph = value.get("@graph")
        if isinstance(graph, list):
            for item in graph:
                yield from iter_nodes(item)
        else:
            yield value

nodes = [node for block in parsed_blocks for node in iter_nodes(block)]
article = next(node for node in nodes if node.get("@type") == "Article")

print("normalized nodes:", len(nodes))
print("selected type:", article["@type"])

@type can also be an array. If your corpus includes publishers that emit "@type": ["Article", "NewsArticle"], normalize that field before testing membership.

Build a Nullable Record

Nested objects need the same care as top-level fields. The author may be a dictionary, a list, a string, or absent. This target uses a dictionary, so the example reads it defensively and preserves missing optional fields as None:

visible_h1 = soup.find("h1").get_text(" ", strip=True)
author = article.get("author") or {}
publisher = article.get("publisher") or {}

record = {
    "type": article.get("@type"),
    "headline": article.get("headline"),
    "visible_h1": visible_h1,
    "author": author.get("name") if isinstance(author, dict) else None,
    "publisher": publisher.get("name") if isinstance(publisher, dict) else None,
    "published": article.get("datePublished"),
    "modified": article.get("dateModified"),
    "image": article.get("image"),
    "description": article.get("description"),
    "keywords": article.get("keywords"),
    "source_url": URL,
}

Keeping source_url in every row makes later audits possible. Without provenance, a corrected parser cannot tell which records need to be rebuilt.

Validate the Fields Your Pipeline Requires

Validation should reflect the downstream contract, not every property Schema.org permits:

required = ("headline", "author", "published", "source_url")
missing = [field for field in required if not record.get(field)]
if missing:
    raise ValueError(f"missing required fields: {missing}")

print("headline:", record["headline"])
print("visible H1:", record["visible_h1"])
print("headings match:", record["headline"] == record["visible_h1"])
print("author:", record["author"])
print("publisher:", record["publisher"])
print("published:", record["published"])

headline: What Is Web Scraping? Definitive Guide 2025
visible H1: What Is Web Scraping? Definitive Guide 2025
headings match: True
author: Emily Chen
publisher: Scrapeless
published: 2025-09-17T08:35:31.224Z

The headline matches on this page. Do not generalize that result: editors sometimes update the visible H1 without updating structured metadata, or use a shorter search headline in JSON-LD. Comparing both surfaces is a useful quality check.

Ready to run the same parser on a rendered response? Open a Scrapeless account and keep the parsing code unchanged.

Complete Runnable Extractor

The complete script joins discovery, normalization, selection, nullable mapping, and validation:

import json
import requests
from bs4 import BeautifulSoup

URL = "https://www.scrapeless.com/en/blog/what-is-web-scraping?utm_source=website&utm_medium=blog&utm_campaign=universalscrapingapi&utm_term=json-ld-structured-data-web-scraping"


def iter_nodes(value):
    if isinstance(value, list):
        for item in value:
            yield from iter_nodes(item)
    elif isinstance(value, dict):
        graph = value.get("@graph")
        if isinstance(graph, list):
            for item in graph:
                yield from iter_nodes(item)
        else:
            yield value


response = requests.get(
    URL,
    headers={"User-Agent": "Mozilla/5.0"},
    timeout=30,
)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")

nodes = []
invalid_blocks = 0
scripts = soup.find_all("script", type="application/ld+json")
for script in scripts:
    try:
        nodes.extend(iter_nodes(json.loads(script.get_text(strip=True))))
    except json.JSONDecodeError:
        invalid_blocks += 1

article = next(node for node in nodes if node.get("@type") == "Article")
author = article.get("author") or {}
publisher = article.get("publisher") or {}
visible_h1 = soup.find("h1").get_text(" ", strip=True)

record = {
    "type": article.get("@type"),
    "headline": article.get("headline"),
    "visible_h1": visible_h1,
    "author": author.get("name") if isinstance(author, dict) else None,
    "publisher": publisher.get("name") if isinstance(publisher, dict) else None,
    "published": article.get("datePublished"),
    "image": article.get("image"),
    "keywords": article.get("keywords"),
    "source_url": URL,
}

required = ("headline", "author", "published", "source_url")
missing = [field for field in required if not record.get(field)]
if missing:
    raise ValueError(f"missing required fields: {missing}")

print(f"HTML bytes: {len(response.content)}")
print(f"JSON-LD blocks: {len(scripts)}")
print(f"normalized nodes: {len(nodes)}")
print(f"invalid blocks: {invalid_blocks}")
print(f"type: {record['type']}")
print(f"headline: {record['headline']}")
print(f"visible H1: {record['visible_h1']}")
print(f"headings match: {record['headline'] == record['visible_h1']}")
print(f"author: {record['author']}")
print(f"publisher: {record['publisher']}")
print(f"published: {record['published']}")
print(f"keyword chars: {len(record['keywords'] or '')}")

The live run returned one valid block, one normalized Article node, no malformed blocks, matching headlines, author Emily Chen, publisher Scrapeless, and 140 characters in the keywords string.

When the JSON-LD Appears Only After Rendering

Beautiful Soup parses the bytes it receives; it does not run a page's JavaScript. A quick diagnostic is to compare the raw response with the browser DOM. If the browser shows an application/ld+json script but requests finds none, fetch rendered HTML before parsing.

Note: The request below requires a funded Scrapeless account. The verification account returned an insufficient-balance response during final checking, so this HTTP call is a prerequisite gap; the parser above ran completely against the real public page.

import os
import requests

rendered = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={
        "actor": "unlocker.webunlocker",
        "input": {
            "url": URL,
            "method": "GET",
            "js_render": True,
        },
    },
    timeout=90,
)
rendered.raise_for_status()
html = rendered.json()["data"]

Pass html into BeautifulSoup and reuse the same normalizer. The Universal Scraping API supplies the rendered response; it does not change the JSON-LD contract.

Common JSON-LD Data Problems

The page has several matching nodes

Select by both type and identity. For product variants, use @id, URL, SKU, or another stable field rather than taking the first Product node.

@type is an array

Convert it to a set before checking membership. A strict equality test against a string will miss a valid multi-type node.

The script contains HTML entities or comments

JSON-LD should be valid JSON text. If a publisher wraps it in invalid syntax, record that block as malformed and fix the parser for that known source; do not apply broad string replacements that can corrupt legitimate values.

Structured metadata disagrees with visible text

Store both values and define precedence for your use case. Search auditing may prefer the JSON-LD headline; content monitoring may prefer the visible H1. A mismatch is data, not merely an error.

A field changes from object to list

Normalize at the boundary. Authors and images commonly switch between one object and an array as a publisher's CMS evolves.

Conclusion

A reliable JSON-LD scraper does four things: collects every matching script, decodes without hiding malformed blocks, normalizes dictionaries, lists, and @graph, then validates a small downstream contract. That path is shorter and usually more stable than rebuilding the same record from page-layout selectors. Keep the visible DOM as a cross-check, retain source provenance, and introduce rendering only when the initial HTML proves it is necessary.

Start with the Scrapeless free plan, review the developer docs, and check Scrapeless pricing before moving a rendered corpus into production.

FAQ

Q: Is JSON-LD easier to scrape than visible HTML?

Yes, when the publisher includes the fields you need. JSON-LD gives you named properties and types, while visible HTML often requires selectors and text cleanup; you should still compare critical fields with the rendered page.

Q: Why should I parse every application/ld+json script?

A page can place different entities in separate blocks. Reading only the first script can return the organization or breadcrumb and miss the article or product you wanted.

Q: What does @graph mean for extraction?

@graph groups several JSON-LD nodes inside one object. Flatten the graph, then select nodes by @type, @id, URL, or another stable identifier.

Q: What if a JSON-LD property is missing?

Keep optional properties as None and fail only when a field required by your own downstream contract is absent. Schema.org describes possible properties; it does not force publishers to fill all of them.

Q: Can JSON-LD differ from the visible page?

Yes. Metadata and visible content can be updated on different schedules or optimized for different surfaces. Store both values when the difference matters and make precedence explicit.

Q: Do I need a browser to extract JSON-LD?

Not when the script is present in the initial HTML. You need rendering only when client-side JavaScript inserts or modifies the structured data after the raw response arrives.

Q: Is extracting public JSON-LD always allowed?

No blanket rule makes every collection lawful or permitted. Review the site's terms and robots directives, keep request volume bounded, collect only the fields you need, and obtain legal advice for sensitive or commercial uses.