Keyword Research from AI Answer Engines: Mine the Answer's Own Outline

TL;DR:

• The AI answer is a ready-made content outline. When an engine answers a topic, the headings and bolded phrases it structures the answer around are the subtopics it considers essential — mine those and you have a keyword and outline map built from what the model actually returns.
• One prompt, several engines, one envelope. The Scrapeless LLM actors (scraper.chatgpt, scraper.gemini, scraper.perplexity, and the rest) share an endpoint and a { status, task_id, task_result } shape, so a single loop captures the answer text from each.
• The signal is the answer's own structure, not its citations. Markdown headings and short bolded phrases in result_text are the subtopics; pulling them needs no model key, just a parser.
• Cross-engine overlap ranks the subtopics. A subtopic that several engines independently raise is one your content almost certainly needs to cover.
• It runs on a schedule. Re-capture a seed topic over time and watch which subtopics the engines start or stop emphasizing.
• Free to start. New Scrapeless accounts include free trial credits — sign up at app.scrapeless.com.

Pipeline at a glance

Traditional keyword research starts from a search box and a volume estimate. AI-answer research starts from the answer itself: ask the engine your seed topic, and read back the structure it imposes — the sections it splits the topic into, the concepts it bolds, the order it puts them in. That structure is a content brief the model wrote for you.

The build is three stages on top of the Universal Scraping API:

1. Capture — run a seed topic across the AI answer engines through their Scrapeless actors; store each answer.
2. Extract — pull the headings and short bolded phrases out of each answer's markdown; those are the candidate subtopics.
3. Rank — count how many engines raise each subtopic; the overlap is your priority order.

The output is a ranked subtopic list you can turn into an outline, a brief, or a keyword cluster. For the companion metric — which sources the engines cite — see the AI Overview scraper guide.

What You Can Do With It

• Build a content brief from the answer. The engine's headings become your H2s; the bolded phrases become the points to cover under each.
• Find gaps in your existing page. Diff the engine's subtopics against the sections you already have, and write what's missing.
• Cluster keywords by intent. Subtopics that co-occur across engines belong in the same piece; ones that stand alone may deserve their own page.
• Track topic drift. Re-capture monthly and watch which subtopics rise — an early read on where a topic is heading.
• Brief writers with evidence. "Three engines structure this around X, Y, and Z" is a stronger brief than a guess.

Why the Scrapeless LLM actors

Each AI assistant is a JavaScript application behind authentication and anti-automation defenses; capturing the answer yourself means rendering, sign-in, and proxy rotation per platform. The Scrapeless LLM actors run that surface server-side and return the answer as a field. For subtopic mining specifically, they bring:

• A shared { status, task_id, task_result } envelope across engines, so one loop and one parser cover the whole set.
• result_text as markdown — the headings and bold markers survive intact, which is exactly what the extractor reads.
• Residential egress in 195+ countries, so a pinned country captures the answer structure a real user in that market sees.
• No browser to run or keep signed in — one endpoint, one x-api-token header.

Pricing for the actor line is usage-based with free trial credits on signup — current tiers are on the pricing page. Get your API key on the free plan at app.scrapeless.com.

Prerequisites

• A Scrapeless account and API key (free plan includes trial credits) — app.scrapeless.com.
• The key in your environment:

export SCRAPELESS_API_KEY="your_api_token_here"

• Python 3 with requests. The extract step uses only the standard library.

Stage 1 — Capture the answers

One loop covers every engine, because the actors share an endpoint and an envelope. The answer text lands in result_text as markdown, headings and bold markers intact.

import json
import os
import time

import requests

ENDPOINT = "https://api.scrapeless.com/api/v2/scraper/execute"
HEADERS = {
    "Content-Type": "application/json",
    "x-api-token": os.environ["SCRAPELESS_API_KEY"],
}

SEED = "web scraping for beginners"
COUNTRY = "US"

ENGINES = {
    "chatgpt": {"actor": "scraper.chatgpt", "extra": {}},
    "perplexity": {"actor": "scraper.perplexity", "extra": {"web_search": True}},
    "gemini": {"actor": "scraper.gemini", "extra": {}},
}

with open("answers.jsonl", "w", encoding="utf-8") as out:
    for platform, spec in ENGINES.items():
        payload = {"actor": spec["actor"], "input": {"prompt": SEED, "country": COUNTRY, **spec["extra"]}}
        data = requests.post(ENDPOINT, headers=HEADERS, json=payload, timeout=300).json()
        result = data.get("task_result") or {}
        out.write(json.dumps({
            "platform": platform,
            "seed": SEED,
            "captured_at": int(time.time()),
            "status": data.get("status"),
            "result_text": result.get("result_text") or "",
        }) + "\n")
        print(f"{platform}: {data.get('status')}")

Each line of answers.jsonl is one engine's full answer for the seed.

Get your API key on the free plan: app.scrapeless.com

Stage 2 and 3 — Extract subtopics and rank by overlap

Pull the headings and short bolded phrases out of each answer's markdown, then count how many engines raised each. The web_results titles are deliberately left out — they carry third-party page names, not subtopics.

# extract.py — answers.jsonl -> ranked subtopic candidates
import json
import re
from collections import Counter

cands = Counter()
for line in open("answers.jsonl", encoding="utf-8"):
    record = json.loads(line)
    text = record["result_text"]
    for heading in re.findall(r"^#{2,4}\s+(.+)$", text, re.M):
        cands[heading.strip().lower()[:60]] += 1
    for bold in re.findall(r"\*\*(.+?)\*\*", text):
        phrase = bold.strip().lower()
        if 2 <= len(phrase.split()) <= 6 and not phrase.startswith("http") and ":" not in phrase:
            cands[phrase[:60]] += 1

ranked = [{"subtopic": k, "hits": c} for k, c in cands.most_common(25) if k]
json.dump(ranked, open("keywords.json", "w"), indent=2)
for item in ranked[:12]:
    print(f'{item["hits"]}x  {item["subtopic"]}')

A live run on the seed "web scraping for beginners" surfaced subtopics like javascript-rendered sites, anti-bot protection, large-scale scraping, the code vs. no-code decision, the python scraping stack, and inspect element — the exact sections a beginner-facing page on the topic should cover. Because the answers regenerate each run, the precise list shifts; the subtopics that recur across engines and across runs are the durable ones to prioritize.

Scheduling and scaling the series

Run capture.py then extract.py on a schedule and append each run keyed by captured_at. A few notes from the live runs:

• Filter the noise. Markdown headings include scaffolding like "what this does" — keep a small stop-list, or require a subtopic to appear in two or more engines before it counts.
• Rank by cross-engine overlap, not raw frequency. A subtopic three engines raise independently is a stronger signal than one engine repeating itself.
• Pin the country. Answer structure shifts by market; keep the value in your records so series stay comparable.
• Pair with citation data. The subtopics tell you what to cover; the citation sources (a separate capture) tell you who the engines currently trust on it.

Conclusion: let the answer write your outline

The fastest content brief for a topic is the one the AI engines already produce every time they answer it. Capture the answer, read its structure, and rank the subtopics across engines, and "what should this page cover?" stops being a guess — it becomes a list you measured.

Ready to Build Your AI-Answer Data Pipeline?

Join our community to claim a free plan and connect with developers building AI-answer pipelines: Discord · Telegram.

Sign up at app.scrapeless.com for free trial credits, and point the pipeline at the seed topics and markets your content program covers.

FAQ

Q: How is this different from a keyword tool?

A: A keyword tool gives you query strings and volume estimates. This gives you the subtopic structure an AI engine imposes on the answer — the sections and concepts it treats as essential — which maps more directly to an outline than a flat keyword list.

Q: Do I need a model API key for the extract step?

A: No. Headings and bold markers are plain markdown, so the extractor uses only the standard library. A model-based pass is an optional upgrade for clustering or labeling.

Q: Why do the subtopics change between runs?

A: Each engine regenerates its answer, so the exact headings vary. That is why the pipeline ranks by cross-engine and cross-run overlap — the recurring subtopics are the stable signal.

Q: Is scraping AI answers legal?

A: The actors read publicly available answer content. As with any scraping, restrict use to public data, respect each platform's terms, avoid personal data, and consult a lawyer if a use case is unclear.