MarkItDown Web Scraping: Convert Pages to LLM-Ready Markdown

TL;DR:

• MarkItDown converts a scraped HTML page into Markdown, keeping the headings, links, and lists that a language model reads well.
• Scraping gives you HTML in memory, so the guide uses convert_stream with a BytesIO and an explicit .html extension rather than a file path.
• MarkItDown has no fetch layer that clears blocks, so Scrapeless retrieves the page and MarkItDown converts it.
• On the example page, 11,021 characters of HTML become 2,973 characters of Markdown that opens with a real heading.
• MarkItDown converts the whole document to structured Markdown; reach for trafilatura instead when you want only the main article with the boilerplate stripped.
• Start on the Scrapeless free plan and convert your first page.

Language models read Markdown better than raw HTML. Markdown carries the structure a model needs, headings, lists, and links, without the tag soup, script blocks, and inline styles that fill an HTML page. MarkItDown, Microsoft's document-to-Markdown converter, does that conversion, and it handles HTML, PDF, Office documents, and more through one interface. What it does not do is fetch the page, which is the half a scraping workflow has to supply.

This guide pairs the two. The Scrapeless Universal Scraping API retrieves the page, and MarkItDown converts the returned HTML to Markdown. Every number below comes from a real run against a public page.

What MarkItDown Does

MarkItDown takes a document and returns Markdown. Its reason for existing is LLM input: it produces text that keeps structure in a token-efficient form, following the widely implemented CommonMark specification for the Markdown it emits. The MarkItDown repository lists the input formats it accepts, which include HTML, PDF, Word, PowerPoint, and images.

What MarkItDown is not is a scraper. It has no browser and no way past an access challenge, so it converts whatever bytes you hand it. Getting those bytes from a live, sometimes-protected page is a separate job.

Install

MarkItDown is a single pip install.

pip install markitdown

Set your Scrapeless key in the shell. Use the real key at run time and keep the placeholder out of your source.

export SCRAPELESS_API_KEY="sk_your_key_here"

Convert Scraped HTML to Markdown

Scraping returns an HTML string, not a file, so the right entry point is convert_stream. Wrap the HTML in a BytesIO and pass file_extension=".html" so MarkItDown parses it as HTML. The converted Markdown is on the result's text_content.

import io
import json
import os
import urllib.request

from markitdown import MarkItDown

API_URL = "https://api.scrapeless.com/api/v2/unlocker/request"
TARGET = "https://quotes.toscrape.com/"


def fetch_html(url: str) -> str:
    payload = json.dumps(
        {"actor": "unlocker.webunlocker", "input": {"url": url, "js_render": False, "headless": False}}
    ).encode()
    request = urllib.request.Request(
        API_URL,
        data=payload,
        headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"], "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=90) as response:
        return json.loads(response.read())["data"]


html = fetch_html(TARGET)
result = MarkItDown().convert_stream(io.BytesIO(html.encode("utf-8")), file_extension=".html")
markdown = result.text_content

print(f"raw html chars: {len(html)}")
print(f"markdown chars: {len(markdown)}")
print(f"starts with heading: {markdown.lstrip().startswith('#')}")
print("--- first lines of markdown ---")
for line in [line for line in markdown.splitlines() if line.strip()][:4]:
    print(line)

The run reports the sizes and prints the top of the Markdown.

raw html chars: 11021
markdown chars: 2973
starts with heading: True
--- first lines of markdown ---
# [Quotes to Scrape](/)
[Login](/login)
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
by Albert Einstein

The output opens with a real Markdown heading and a link, and the page's first quote follows. Structure the model can use survives the conversion, and the 11,021 characters of HTML come down to 2,973 characters of Markdown.

Why Markdown for an LLM

Markdown is the format most language models are trained to read, so headings become sections, links stay legible, and lists remain lists, all in far fewer tokens than the original HTML. Passing raw HTML instead spends tokens on tags and inline styles the model has to see past, and it buries the structure that helps the model organize the content. Converting to Markdown first is a cheap step that pays off on every downstream call.

Where MarkItDown Stops

MarkItDown converts; it does not retrieve past a block, which is why this guide pairs it with a retrieval tool. Hand MarkItDown the HTML from a protected page fetched with a plain client and it faithfully converts a challenge page into Markdown. Scrapeless returns the real rendered HTML, and MarkItDown converts that. When a target builds its content with JavaScript, set js_render to True in the request so the HTML already contains the content; leave it False, as here, when the markup is server-rendered.

MarkItDown and trafilatura solve different problems. MarkItDown converts the whole document to structured Markdown, keeping the page's shape. Trafilatura isolates the main article and drops the boilerplate. Convert with MarkItDown when you want the page as Markdown; extract with trafilatura when you want only the article body.

Before running this across a site, read its robots.txt and terms. The Robots Exclusion Protocol states which paths a site asks automated clients to avoid, and honoring it keeps a conversion pipeline sustainable. For the broader pattern, the guide on AI content pipelines shows where a conversion step like this fits.

Ready to convert your own pages? Create a free Scrapeless account and change the target URL.

Conclusion

Feeding a model Markdown instead of HTML is a small change with a real payoff, and it takes two tools: Scrapeless to retrieve the page and MarkItDown to convert it. One convert_stream call turns 11,021 characters of HTML into 2,973 characters of structured Markdown, ready for an embedding step or a prompt. Start from the script above, point it at your own URL, and pass the Markdown to your model.

Start with the Scrapeless free plan to retrieve your own pages, and check Scrapeless pricing when you size a recurring job.

FAQ

Q: Does MarkItDown fetch web pages by itself?

No. MarkItDown converts documents you give it and has no browser or unblocking, so on a protected page it would convert a challenge page rather than the content. Pair it with a retrieval tool such as Scrapeless that returns rendered HTML, then convert that HTML.

Q: How do I pass scraped HTML to MarkItDown without saving a file?

Use convert_stream with the HTML wrapped in io.BytesIO and file_extension=".html", so MarkItDown parses the in-memory bytes as HTML. This avoids writing the scraped page to disk just to read it back, and the Markdown is on the result's text_content.

Q: What formats can MarkItDown convert?

MarkItDown accepts HTML, PDF, Word, PowerPoint, Excel, images, and several other formats, converting each to Markdown through the same interface. For web scraping the relevant input is HTML, but the same converter handles a scraped PDF or spreadsheet if your pipeline collects those too.

Q: Should I use MarkItDown or trafilatura?

Use MarkItDown when you want the whole document converted to structured Markdown, and trafilatura when you want only the main article with navigation and footers removed. They solve different problems: one is a format converter, the other is a main-content extractor.

Q: Why convert to Markdown before calling a model?

Markdown keeps headings, lists, and links in a compact form language models read well, while raw HTML spends tokens on tags and styles the model must ignore. Converting first cuts token cost and gives the model cleaner structure to work with.