Observe SERP Features Without Mixing Them with Organic Rankings

TL;DR:

• SERP features analysis needs separate measurement units. Organic positions, knowledge graph fields, ads, and related-search observations should not be merged into one ranking list.
• Inventory the actual response before writing module adapters. A top-level JSON field is not automatically a visible search feature, and an omitted field does not prove absence on the page.
• Keep state with every measurement. Pending collection, unmapped fields, nulls, empty containers, and observed values deserve distinct records.

A report that places an organic link, a knowledge panel, and an advertisement in one ranked list loses the meaning of every position it contains. Those elements serve different roles, and their data can have different shapes. Counting them together may create a tidy spreadsheet while making comparisons harder to interpret.

Scrapeless Google Search API supplies structured search data that can support SERP features analysis. The first implementation task is to inspect what a completed response actually contains. This guide builds a local field inventory, then explains how to map verified modules without changing the meaning of organic rankings.

Define the Unit for Each Observation

Start with the question the report should answer. Organic URL presence asks which web links appeared in the collected organic array. A knowledge graph observation asks what structured entity information was supplied. An advertising observation asks about a separately identified paid-search element.

Give each question its own record type and denominator. An array item's ordinal is source order. A returned organic position belongs to the organic result schema. Neither can be reused as an advertisement's position or a panel's visual placement without a separately verified mapping.

Keep the query, complete request, observation time, and collection state common across these records. That shared context lets analysts compare modules from the same run without flattening them into a single unit.

Related-search suggestions, when available through a verified field, belong to query exploration. They are not ranked organic pages. Store their own observed text and provenance rather than assigning the next organic position after the last web link.

Prerequisites and Documented Response Shapes

Use Python and a saved JSON capture for the local inventory. The outer request, http_status, response, run_id, and received_at fields are collector metadata. The code does not send requests or establish account-level feature coverage.

The Google Search request workflow documents the actor scraper.google.search, POST https://api.scrapeless.com/api/v1/scraper/request, and the x-api-token header. Live collection needs your account key. The Google Search parameters explain the search context that should accompany every observation.

The current quickstart example includes an organic_results array, a knowledge_graph object, and a local_results object. It also contains non-module information such as metadata and pagination. These are documentation examples, not results collected for this article.

Note: An authenticated API request was not run here. The local inventory was tested on synthetic captures. Ads and related-search adapters are intentionally left unconfigured because this quickstart example does not establish their field names or complete response contracts. Verify those mappings against current documentation and actual account output before collecting them.

Separate Transport Outcome From Field State

Check the HTTP outcome before inspecting module content. HTTP 201 represents a pending task; HTTP 200 carries task data. A pending response should not generate an “all features absent” observation.

Within a completed response, preserve the difference between a missing key, a null value, an empty array, and an empty object. The JSON value types make those distinctions explicit. Your analytical model should retain them until a verified adapter assigns a narrower meaning.

An empty object may be a structural placeholder. A nonempty object may contain only empty nested values. Neither condition establishes that a person saw a useful knowledge panel on the rendered page. The inventory below reports JSON structure; a module-specific adapter must inspect the relevant nested fields.

Run a Local Field Inventory

Save the program as module_inventory.py and run python3 module_inventory.py capture.json. It prints a derived inventory and leaves the input file unchanged. Python's JSON parser and serializer handle the input and output using standard-library functions.

import argparse
import json
from pathlib import Path


def inventory(record):
    if not isinstance(record, dict):
        raise ValueError('Capture must be an object')
    status, payload = record.get('http_status'), record.get('response')
    base = {'run_id': record.get('run_id'), 'request': record.get('request'),
            'received_at': record.get('received_at'), 'fields': []}
    if status == 201:
        return dict(base, state='pending')
    if status != 200:
        return dict(base, state='transport_error' if status is None else 'http_error')
    if not isinstance(payload, dict):
        return dict(base, state='unmapped')
    fields = []
    for key, value in payload.items():
        if value is None:
            kind, state, size = 'null', 'null', None
        elif isinstance(value, list):
            kind, state, size = 'array', 'nonempty' if value else 'empty', len(value)
        elif isinstance(value, dict):
            kind, state, size = 'object', 'nonempty' if value else 'empty', len(value)
        else:
            kind = 'boolean' if isinstance(value, bool) else ('string' if isinstance(value, str) else 'number')
            state, size = 'scalar', None
        fields.append({'key': key, 'json_type': kind, 'field_state': state, 'size': size})
    return dict(base, state='inventoried', fields=fields)


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('capture')
    args = parser.parse_args()
    print(json.dumps(inventory(json.loads(Path(args.capture).read_text(encoding='utf-8'))),
                     ensure_ascii=False, indent=2))

For arrays, size is the number of array elements. For objects, it is the number of immediate keys. It is not a result count, a panel count, or a measure of visible content. Scalars and nulls have no size in this application schema.

Missing keys do not appear in this raw inventory. Compare the inventory with an explicit expected-field list only after you have established the response contract for the chosen query surface. Otherwise, a guessed list can manufacture apparent missing features.

Start Scraping with Scrapeless

Power up your web scraping and automation workflow with Scrapeless!
Sign up today and get $5 in free credit — no credit card required.

Claim your free credit now in the Scrapeless Dashboard.

Map Verified Fields Into Separate Datasets

Create an organic dataset containing the run reference, item ordinal, returned position, URL, title, and snippet when available. Validate the organic array's shape before interpreting its row count. Preserve unsupported values in the raw capture and flag the mapping issue.

For a knowledge graph dataset, store the raw object reference and individually verified fields. The example response's object shape is a starting point for inspection, not a promise that every entity property is always populated. Avoid a Boolean panel_present derived only from the key's existence.

An ads dataset and a related-search dataset can be reserved in your application design, but leave their adapters disabled until field names, shapes, and state semantics are verified. A disabled adapter should report not_configured, not zero observations. That label describes your application, not API support.

Keep schema version and adapter version beside the derived record. If a field changes, the original capture remains available for a new mapping. The JSON Schema object model can help formalize required and optional properties without treating every missing optional property as a collection failure.

Compare Like With Like Across Runs

A comparison needs matching query scope, country, language, and collection rules. Deliberate changes belong in the report. A country change can alter the search context; it should not be mistaken for a feature change caused by your website.

Compare organic positions only within the same defined organic observation method. Compare module states within the same adapter version and verified field semantics. When a mapper changes, either reprocess both captures or state that the records are not directly comparable.

Separate sample size from collection coverage. A pending run does not join the denominator for a module-presence rate. An unmapped field may require a separate unknown category. Explain the denominator so a percentage, if your team calculates one, has an inspectable meaning.

Preserve evidence for a change with the previous run, current run, raw field values, mapper version, and review outcome. The provenance model is useful for separating the observation from the transformation that created the report.

Review What the Data Cannot Establish

A JSON response does not automatically establish pixel placement, viewport visibility, or the amount of screen space a feature occupied. Those questions need a separately captured and verified visual method. Do not describe field order as a screenshot coordinate.

Likewise, a module observation does not prove a click, visit, conversion, or impression count. Keep those business outcomes in their own data sources. A search feature can be relevant to a research question without becoming a proxy for every downstream outcome.

Inspect unexpected states before writing an alert. A null field, disabled adapter, or changed response shape should lead to a data review. Only a comparable, interpreted observation can support a claim that a feature changed within the collected sample.

Conclusion

Inventory the response, verify each module's contract, and store organic rankings separately from other search features. Explicit field states and adapter versions make the resulting report easier to review and prevent missing data from turning into an invented feature trend.

Use reviewed module observations as one input to content gap analysis when deciding which search experiences deserve further editorial research.

Build Your Next Search Observation

Use Scrapeless Google Search API to collect the search evidence for this workflow. Review Scrapeless pricing when planning your collection budget, and keep the Google Search parameters beside your request configuration.

Discuss your implementation with the community on Discord or Telegram.

FAQ

Q: Does a knowledge_graph key prove that a populated panel appeared?

No. Inspect the value and verified nested fields. Key presence and visible populated content are different claims.

Q: Can ads be appended after organic results in one ranking list?

That would mix measurement units. Preserve separate datasets and only use positions whose meanings are established by the relevant schema.

Q: Does the inventory count search features?

No. It records top-level JSON types and container sizes. Feature-specific meaning requires a verified adapter.

Q: What does an omitted field mean?

It means the key was absent from the captured object. It does not, by itself, establish that the corresponding feature was absent from the rendered search page.

Q: Are ads and related searches implemented in the sample?

No. Their adapters remain unconfigured pending verification of the appropriate fields and actual account output. The article does not invent those mappings.