How to Build an AI Training-Data Pipeline With Scrapeless

TL;DR:

• A fine-tuning dataset is a pipeline product, not a download. This guide builds one end to end: fetch rendered public pages through the Scrapeless Universal Scraping API, extract labeled pairs with the Python standard library, and write OpenAI-ready chat-format JSONL with a train/validation split.
• The fetch layer is one HTTP POST per page. The unlocker.webunlocker actor returns rendered HTML from POST /api/v1/unlocker/request — no browser to manage and no proxy pool to rotate on your side.
• Everything up to the upload runs with a Scrapeless key alone. The demo crawl covers all 10 pages of a public quotes site and produces 100 supervised examples; only the final fine-tune job needs an OpenAI key and budget.
• Format errors are the cheapest ones to prevent. Each training line is a {"messages": [...]} object with system, user, and assistant turns — write it with json.dumps and the file parses on the first upload.
• Provenance is part of the dataset. Public pages only, bounded volume, and site terms and robots directives checked before the crawl — the responsibility section covers what training data adds to the usual scraping questions.
• Free to start. Create your API key on the free plan at app.scrapeless.com.

Introduction: the dataset is the hard part

Fine-tuning an LLM is administratively easy — upload a file, create a job, wait. What the job cannot fix is the file. A model tuned on a hundred well-formed, correctly labeled examples of the task can beat one tuned on ten thousand noisy lines, and that difference is decided before the training run starts, in the pipeline that collected and shaped the data.

That pipeline is a scraping problem for most teams, because the domain knowledge worth tuning on lives on web pages: product descriptions, documentation, listings, reviews, reference material. The conceptual side of the journey is covered in the guide to how AI model training works end to end; this post is the executable half. You will build a complete pipeline against a public demo site — quotes.toscrape.com, a site built for scraping practice — and finish with train.jsonl and val.jsonl files that the OpenAI fine-tuning endpoint accepts as-is.

The demo task: teach a model to attribute famous quotes. Small enough to run in minutes, structured exactly like the real thing.

Pipeline at a Glance

The pipeline is five stages, and the first four run for real in this guide with only a Scrapeless API key:

1. Fetch — retrieve each rendered page through the Universal Scraping API, one POST per page.
2. Discover — follow the site's own pagination until it ends, with a hard page cap as a safety bound.
3. Extract — parse quote text and author out of each page with Python's standard-library HTML parser.
4. Transform — turn each pair into a chat-format training example, split 80/20, and write JSONL.
5. Train — upload both files and create the fine-tune job (this stage needs an OpenAI key; the code is shown and labeled accordingly).

Flow per page: render or fetch → discover next page → extract pairs → transform to examples → store as JSONL.

Why Scrapeless Universal Scraping API

The Universal Scraping API turns page collection into a deterministic function call: you POST a URL, the service handles rendering, unlocking, and proxy routing server-side, and you get the page HTML back in the data field. For a dataset builder that matters twice over. First, the corpus stays reproducible — the same request shape works whether the source is a static demo site or a JavaScript-heavy product catalog, so the pipeline code does not change when the target does. Second, there is no local browser fleet: a dataset of a few hundred pages is a loop over one function.

The demo site here is deliberately friendly. The point of the guide is the pipeline shape; swap the URL list and the parser when you point it at your real corpus, and keep the volume bounds.

Prerequisites

• Python 3 with the requests package — every other import in the pipeline is standard library; the runs in this guide used Python 3.12.
• A Scrapeless API key — the developer docs cover key creation.
• For Stage 5 only: an OpenAI API key with fine-tuning access and budget. Stages 1–4 run without it.

Export the Scrapeless key so the scripts read it from the environment:

export SCRAPELESS_API_KEY="sk_your_key_here"

Stage 1 — Fetch rendered pages

One POST returns one rendered page. The unlocker.webunlocker actor takes the target URL in input and answers with the HTML in the response's data field:

# fetch_page.py — Stage 1: retrieve one rendered page through Scrapeless
import os

import requests

ENDPOINT = "https://api.scrapeless.com/api/v1/unlocker/request"
HEADERS = {
    "Content-Type": "application/json",
    "x-api-token": os.environ["SCRAPELESS_API_KEY"],
}


def fetch_page(url: str) -> str:
    resp = requests.post(
        ENDPOINT,
        headers=HEADERS,
        json={"actor": "unlocker.webunlocker", "input": {"url": url, "method": "GET"}},
        timeout=120,
    )
    resp.raise_for_status()
    return resp.json().get("data", "")


html = fetch_page("https://quotes.toscrape.com/page/1/")
print(f"fetched {len(html):,} characters of HTML")
print("quote markup present:", '<span class="text"' in html)
print("author markup present:", '<small class="author"' in html)

Against page 1 of the quotes site this returns 11,021 characters of HTML with both markers present. raise_for_status() keeps failures loud: a bad key or an unreachable target stops the pipeline instead of writing an empty corpus.

Stage 2 — Discover the full corpus

The site tells you where it ends, so the crawler follows the site instead of guessing URLs. Each page of the demo site carries a li class="next" element until the last one; the crawl loop fetches, checks for that marker, and advances the page counter. Two bounds keep the stage honest: the loop stops when the marker disappears, and a MAX_PAGES cap stops it even if the marker never does. The cap is the difference between a dataset build and an unbounded crawl — set it to the corpus size you actually intend to collect.

The loop itself is four lines inside the full pipeline script in Stage 4.

Stage 3 — Extract quote–author pairs

Extraction turns HTML into labeled pairs, and the standard library is enough for it. html.parser.HTMLParser fires callbacks per tag; tracking two flags while walking the page collects each quote's text and author without any third-party dependency:

# quote_parser.py — Stage 3: stdlib extraction of (quote, author) pairs
from html.parser import HTMLParser


class QuoteParser(HTMLParser):
    """Collect (text, author) pairs from quotes.toscrape.com markup."""

    def __init__(self):
        super().__init__()
        self.pairs, self._text, self._mode = [], "", None

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "span" and a.get("class") == "text":
            self._mode = "text"
        elif tag == "small" and a.get("class") == "author":
            self._mode = "author"

    def handle_data(self, data):
        if self._mode == "text":
            self._text = data.strip("“”")
        elif self._mode == "author":
            self.pairs.append((self._text, data.strip()))
        self._mode = None


def parse_quotes(html: str):
    p = QuoteParser()
    p.feed(html)
    return p.pairs

A selector library would work too — the reason to show the stdlib version is that the whole pipeline stays a two-dependency script (requests plus Python itself), which is one less thing to pin when the pipeline moves to a scheduler.

Stage 4 — Transform to chat-format JSONL

The OpenAI fine-tuning endpoint trains on chat transcripts: each line of the file is one {"messages": [...]} object with the system rule, the user's input, and the assistant answer you want the model to learn. The format is JSON Lines — the JSON Lines format definition is exactly "one JSON value per line" — and research on instruction tuning such as the InstructGPT paper is the reason the shape looks like a conversation: models follow tasks better when trained on demonstration pairs.

This script is the whole pipeline — Stages 1 through 4 composed, ending in two files:

# build_dataset.py — Stages 1–4: crawl, extract, transform, store
import json
import os

import requests
from html.parser import HTMLParser

ENDPOINT = "https://api.scrapeless.com/api/v1/unlocker/request"
HEADERS = {
    "Content-Type": "application/json",
    "x-api-token": os.environ["SCRAPELESS_API_KEY"],
}
BASE = "https://quotes.toscrape.com"
MAX_PAGES = 15  # safety bound above the site's real size
SYSTEM = "You attribute famous quotes. Reply with only the author's name."


class QuoteParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.pairs, self._text, self._mode = [], "", None

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "span" and a.get("class") == "text":
            self._mode = "text"
        elif tag == "small" and a.get("class") == "author":
            self._mode = "author"

    def handle_data(self, data):
        if self._mode == "text":
            self._text = data.strip("“”")
        elif self._mode == "author":
            self.pairs.append((self._text, data.strip()))
        self._mode = None


def fetch_page(url: str) -> str:
    resp = requests.post(
        ENDPOINT,
        headers=HEADERS,
        json={"actor": "unlocker.webunlocker", "input": {"url": url, "method": "GET"}},
        timeout=120,
    )
    resp.raise_for_status()
    return resp.json().get("data", "")


def to_example(text: str, author: str) -> dict:
    return {
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"Who said this: “{text}”"},
            {"role": "assistant", "content": author},
        ]
    }


pairs, page = [], 1
while page <= MAX_PAGES:
    html = fetch_page(f"{BASE}/page/{page}/")
    parser = QuoteParser()
    parser.feed(html)
    pairs.extend(parser.pairs)
    if 'class="next"' not in html:  # Stage 2: the site says when it ends
        break
    page += 1

examples = [to_example(t, a) for t, a in pairs]
split = int(len(examples) * 0.8)
for name, rows in (("train.jsonl", examples[:split]), ("val.jsonl", examples[split:])):
    with open(name, "w", encoding="utf-8") as f:
        f.writelines(json.dumps(r, ensure_ascii=False) + "\n" for r in rows)

print(f"pages crawled: {page} | pairs extracted: {len(pairs)}")
print(f"train.jsonl: {split} examples | val.jsonl: {len(examples) - split} examples")
print("first training line:")
print(json.dumps(examples[0], ensure_ascii=False)[:180])

The run crawls all 10 pages of the site, extracts 100 pairs, and writes 80 training examples against 20 validation examples. The 80/20 split gives the fine-tuning job something to measure generalization on — validation examples the model never trains on. Both files are written with ensure_ascii=False and one json.dumps per line, which is the cheapest possible insurance: malformed JSONL is a common way to lose a round trip to the upload endpoint.

One quality pass is worth doing by hand even on a toy corpus: read a sample of lines and confirm the label really answers the input. A model learns whatever the file demonstrates, including the mistakes.

Stage 5 — Submit the fine-tune job

The training call is small compared to everything above it. Upload both files with purpose fine-tune, then create the job against a fine-tunable model — the current list and parameters are in the OpenAI supervised fine-tuning guide.

Note: This stage is the pipeline's one prerequisite gap — it needs an OPENAI_API_KEY with fine-tuning access and budget, which this guide does not assume. Everything above ran for real with only a Scrapeless key.

# submit_job.py — Stage 5: upload the dataset and create the job (requires OPENAI_API_KEY)
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment

train = client.files.create(file=open("train.jsonl", "rb"), purpose="fine-tune")
val = client.files.create(file=open("val.jsonl", "rb"), purpose="fine-tune")

job = client.fine_tuning.jobs.create(
    training_file=train.id,
    validation_file=val.id,
    model="gpt-4.1-mini-2025-04-14",
)
print(job.id, job.status)

When the job finishes, the resulting model id drops into the same chat-completions call you already use — the dataset decides whether that model actually answers "Albert Einstein" when shown a quote it has never seen.

Get your API key on the free plan: app.scrapeless.com

Scraping responsibly for training data

Training data carries every obligation the source pages carried, plus one more: the model will reproduce patterns from whatever you feed it. Four practices keep the collection side defensible.

• Public pages only, and read the terms. Collect only what renders without an account, and check the target site's terms of service before the crawl — training use is called out explicitly by a growing number of sites.
• Honor robots directives. the Robots Exclusion Protocol (RFC 9309) is the standard machine-readable statement of what a site allows crawlers to touch; check it for your target paths before collecting.
• Minimize. Take the fields the task needs — here, quote text and author — not whole-page dumps that drag in user comments, names, or other incidental data. Bounded page caps are part of minimization.
• Track provenance and licensing. Record where each example came from and when. Text that is public to read is not automatically licensed for model training in every jurisdiction; when the corpus is anything more sensitive than famous quotes, have counsel look at the licensing question before the job runs.

What You Get Back

Two files, ready for upload. Each line is one complete supervised example — the first line of train.jsonl from the run above:

{"messages": [{"role": "system", "content": "You attribute famous quotes. Reply with only the author's name."}, {"role": "user", "content": "Who said this: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”"}, {"role": "assistant", "content": "Albert Einstein"}]}

The shape scales without changing: a real corpus swaps the parser and the URL list, the system rule describes your task instead of quote attribution, and the JSONL writer, the split, and the upload stay identical. If your goal is retrieval rather than weight updates, the same fetch layer feeds the clean-text RAG pipeline instead — chunks and embeddings rather than training examples.

Conclusion

The pipeline earns its keep at the two ends. At the front, the Universal Scraping API makes collection a deterministic POST per page, so the corpus is reproducible and the code survives a change of target. At the back, disciplined transformation — exact chat format, one JSON object per line, a real validation split, a human pass over the labels — is what separates a fine-tune that improves answers from one that burns budget. Between those ends, the middle is a hundred lines of standard-library Python you now have.

Ready to Build Your Training-Data Pipeline?

Plans and included request volumes are on the pricing page, and the fetch layer in this guide works on the free plan — create your API key at app.scrapeless.com and Stage 1 returns your first rendered page in one POST.

FAQ

Q: How much data do I need to fine-tune a model?

Less than most teams expect, if the examples are clean. Supervised fine-tuning shows measurable behavior change with corpora in the tens to hundreds of well-labeled examples for narrow tasks; broad-behavior changes need more. Start with the smallest corpus that represents the task, evaluate against the validation file, and grow the dataset where the model actually fails.

Q: What format does the fine-tuning endpoint expect?

Chat-format JSON Lines: each line is a standalone JSON object with a messages array of system, user, and assistant turns, uploaded with purpose fine-tune. The pipeline in this guide writes that format directly with json.dumps, one object per line, no trailing commas or wrapping array.

Q: Should I fine-tune or use RAG?

Fine-tune when you want the model to change behavior — tone, format, task-specific reflexes — and retrieval when you want it to know current facts. Weight updates bake in patterns but go stale; retrieval stays current but does not teach the model new habits. The two compose, and both start from the same collection layer this guide builds.

Q: Is it legal to train a model on scraped data?

It depends on what you collect and where you operate, and reading public pages is not automatically a license to train on them. Site terms, robots directives, copyright, and the privacy laws that cover any personal data in the corpus all apply — the responsibility section above lists the working practices, and for anything beyond clearly public, non-personal content the licensing question belongs with counsel.

Q: Does this pipeline work for open-weight models too?

Yes — the collection and transformation stages are model-agnostic. Chat-format JSONL is the common denominator across tuning stacks; open-weight workflows consume the same conversation structure, so the only stage that changes is Stage 5, where the upload call is replaced by your training framework's dataset loader.