Compare Competitor Domains in Google Organic Results

TL;DR:

• Competitor SERP analysis measures presence in a selected query sample. State the queries, context, collection window, and result depth beside every comparison.
• Use hostname rules and count observations consistently. Several URLs from one domain should not accidentally become several independent wins.
• Inspect the pages behind a difference. Organic-result presence can guide content research, but it does not establish traffic, revenue, or advertising spend.

A company competing for the same customers may not appear for every query your team cares about. A publisher, directory, or discussion forum may appear instead. Search competitors are defined by the search sample, so the comparison should begin with the queries rather than a sales team's competitor list alone.

Competitor SERP analysis uses Scrapeless Google Search API to collect structured observations, then asks which domains appear and which pages serve the apparent intent. This guide defines the comparison rules before the scorecard, so the result can support a content decision without implying a market-share measurement.

Select Queries That Represent a Decision

A query set should represent a documented research question. Group phrases by task, such as learning a concept, comparing a category, or finding a specific product capability.

Keep exact wording and topic labels separately. Related phrases may be useful members of the same topic, but they are distinct observations. Repeating a near-identical query many times can give that wording excessive influence in a simple average.

Record why each query was included. Use a stable query-set revision for repeated reporting. If the team adds new topics, show the new coverage separately or recalculate a clearly labeled common subset for comparison.

The sample may be valuable without being statistically representative of all demand. Avoid calling its domain-presence percentage a total search-market share. Search volume is not supplied by this workflow, so equal query weighting is a design choice rather than a demand estimate.

Define the Domain Grouping Rule

Domain matching should use parsed hostnames and explicit boundaries. Matching a string anywhere in a URL can count unrelated sites, path text, or query parameters as a competitor.

Preserve the original hostname and decide whether to include subdomains. A rule that accepts the exact target hostname or a hostname ending in a dot plus that target has a clear boundary. It still needs the target to be a verified organization domain; a public suffix is not a valid organization target.

The public-suffix boundary matters when a report groups hostnames into registrable domains. Taking the last two labels is not a general solution. Some suffixes contain several labels, and hosted platforms may separate unrelated publishers under a shared parent.

Use a reviewed allowlist when comparing a small set of organizations. Store the policy version so a later mapping change does not masquerade as a change in search presence. Keep unknown and unparseable links visible for review rather than silently assigning them to a competitor.

Collect Comparable Organic Slices

The Google Search input parameters let the collector set query, country, language, and pagination context. Keep those settings fixed within a comparison and save the complete request.

The structured request workflow uses POST https://api.scrapeless.com/api/v1/scraper/request, the x-api-token header, and actor scraper.google.search. Authenticated collection requires an account key. No live competitor dataset or measured competitor performance is claimed in this article.

Only map organic results after confirming the response state and array shape. HTTP 201 is pending; a failed or unmapped run cannot support an absence claim. A present empty array should be retained with its own state so reviewers can distinguish it from missing collection.

Store returned positions, titles, links, and snippets when present. Keep other modules in their original response areas. The JSON structure is useful evidence even when a downstream report reduces results to domain-level rows.

Build Metrics With Explicit Denominators

A simple domain-presence measure counts usable observations in which at least one organic URL matches the domain. Divide by the number of usable observations in the stated comparison group. Count the domain once per observation, even if several of its pages appear.

Show the numerator and denominator next to any percentage. Also show how many planned observations were not usable. Otherwise a collection gap can change the denominator without the reader noticing.

A best-returned-position measure can take the smallest valid returned organic position for a matched domain within an observation. Keep missing positions null. Do not assign an arbitrary position to an absent domain and then average that invented number with observed positions.

Domain overlap answers another question: which domains appear in both selected samples? It can help identify shared publishers, but it discards order and page identity. Report it alongside page-level evidence when the team's next step is content research.

Start Scraping with Scrapeless

Power up your web scraping and automation workflow with Scrapeless!
Sign up today and get $5 in free credit — no credit card required.

Claim your free credit now in the Scrapeless Dashboard.

Separate Domain Presence From Page Competition

A domain-level comparison can hide different page strategies. One organization may appear through a category page, another through a tutorial, and a third through a discussion thread.

Open the matched URLs and label page purpose after review. Record whether the page answers the query's task, which audience it addresses, and which substantive questions it covers. A title or snippet is a starting point for that review, not a substitute for the content.

Keep the exact URL alongside a reviewed canonical or normalized grouping if your analysis uses one. Google describes canonical URL selection as a separate search process. A locally normalized URL is your analytical choice and should not be presented as Google's selected canonical unless you have that evidence.

When several pages from your own domain alternate, inspect the pages before labeling the pattern as a problem. The queries or intended audiences may differ. A presence chart can identify where to look; it cannot determine the correct editorial action by itself.

Write a Finding That Can Be Checked

A defensible finding names the sample and the observation. For example, a report can say that a reviewed domain appeared in the usable organic slices for a particular topic group during the collection window. It should link that statement to the contributing runs.

The next sentence can propose an investigation: review the competitor's page types, compare the questions answered, or check whether your own content addresses the same task. Label that proposal as an interpretation rather than a measured cause.

Avoid converting a result-position difference into expected revenue or advertising spend. Organic search observations contain neither business outcome. They also do not provide a complete inventory of a competitor's indexed pages; site-restricted search coverage has its own limits.

If stakeholders want traffic analysis, bring in a separately defined data source and explain its scope. Keep its estimates distinct from directly collected search observations.

Maintain the Comparison Over Time

Repeated reporting needs stable definitions as well as stable query inputs. Version the query set, domain allowlist, parser, and metric rules.

Use common usable observations when directly comparing two periods, and disclose excluded runs. A separate coverage report can show the full collection plan. That prevents a convenient denominator from hiding missing markets or topics.

Keep page-review notes and business actions outside the raw data. If the team rewrites a page, record the action time and rationale. Later search movement can motivate further investigation, but temporal sequence alone does not establish that the edit caused the change.

Archive the evidence behind each report. A reviewer should be able to locate the original request, raw response, normalized rows, grouping policy, and written interpretation without reconstructing the analysis from a screenshot.

Conclusion

Define the query sample and domain rules before calculating presence. Preserve collection coverage and use page review to explain differences. That produces a competitor comparison your team can act on while keeping the limits of the search evidence clear.

Use reviewed page findings to inform content gap analysis after the domain comparison identifies where to investigate.

Build Your Next Search Observation

Configure Google Search API around the questions your team needs to answer. Check Scrapeless pricing before setting collection frequency. The Google Search parameter model explains the context controls used in this workflow.

Discuss your implementation with the community on Discord or Telegram.

FAQ

Q: Does competitor SERP analysis estimate competitor traffic?

This workflow does not. It measures observed organic presence within the selected query sample. Traffic needs a separately defined source.

Q: Should several URLs from one domain count several times?

For domain presence, count the domain once per observation. A page-level report may retain every URL, but that is a different measure.

Q: Can the last two hostname labels identify every organization?

No. Public suffixes and hosting arrangements make that rule unreliable. Use a reviewed domain policy and preserve the original hostname.

Q: Is a domain absent from a collected page absent from Google?

No. The observation covers only the requested and successfully collected slice. Broader absence claims require different evidence.

Q: Can the scorecard prove why a competitor ranks?

No. It identifies patterns for investigation. Review content and other relevant evidence before proposing an explanation, and keep causal claims separate from observations.