How to Build a News Crawler: A Step-by-Step Guide

TL;DR:

• A news crawler is two loops: discover, then fetch. Pull article links from a section front page, then visit each link and extract the headline and body — keep the two stages separate so each can be re-run, paced, and scaled on its own.
• Discovery is a link-filter problem. A front page is mostly navigation; the article links match a recognizable URL shape (/news/articles/…, a slug ending in an id). Filter to those instead of grabbing every anchor.
• Extraction means the readable core, not the whole DOM. Read the h1 and the article paragraphs; skip nav, related-links, and ad rails. A paragraph count and text length tell you instantly whether the fetch actually got the story.
• News sites geo-route and rate-limit. Pin proxyCountry and pace the crawl so a section sweep doesn't trip anti-bot limits halfway through.
• It runs on Scrapeless Scraping Browser with plain Puppeteer. The cloud browser renders JS-built front pages and supplies residential egress; your crawler code is ordinary navigation and extraction.
• Free to start. New Scrapeless accounts include free Scraping Browser runtime — sign up at app.scrapeless.com.

Introduction: what a news crawler actually does

A news crawler turns a section front page into a stream of structured articles. It does two distinct jobs. First, discovery: read the front page (or a section, or a sitemap) and collect the links that point at actual stories. Second, fetching: visit each story and pull out the headline and the readable body, dropping the navigation and the ad furniture.

Keeping those two stages separate is the whole trick. Discovery runs once per sweep and is cheap; fetching runs once per article and is where the real work — and the rate-limit risk — lives. Separating them lets you collect a clean link list first, then fetch at a controlled pace, and re-fetch a single article without re-crawling the front page.

This guide builds the crawler in Node on Scrapeless Scraping Browser — an anti-detection cloud browser connected to Puppeteer over a standard endpoint. The discovery and extraction steps below were both run against live news sources. Public content only.

What You Can Do With It

• Sweep a section (world, business, tech) for the day's new articles.
• Extract clean article bodies — headline plus paragraphs — for analysis or a reading pipeline.
• Feed a summarizer or RAG store with the readable text instead of raw HTML.
• Track a beat by re-running discovery on a schedule and fetching only new links.
• Cross-source by pointing the same crawler at several outlets.

Why Scrapeless Scraping Browser

Scrapeless Scraping Browser is a customizable, anti-detection cloud browser designed for web crawlers and AI agents. For a news crawler specifically, it brings:

• Cloud-side JS rendering — modern front pages build their link lists with JavaScript; the cloud browser runs it so the links exist.
• Residential proxies in 195+ countries — pin egress to the region a site serves, and avoid the rate-limiting a section sweep can trigger.
• Anti-detection fingerprinting — the crawl reads as a real browser, so article pages keep rendering across a long run.
• A standard Puppeteer connection — Puppeteer.connect() returns a normal Browser; discovery and extraction are plain Puppeteer.
• Session persistence — keep one session warm across the whole sweep.

Get your API key on the free plan at app.scrapeless.com.

Prerequisites

• Node.js 18 or newer
• A Scrapeless account and API key — sign up at app.scrapeless.com
• Basic familiarity with Puppeteer

Install

npm install @scrapeless-ai/sdk puppeteer-core

export SCRAPELESS_API_KEY="your_api_token_here"

Step 1 — Connect

import { Puppeteer } from '@scrapeless-ai/sdk';

const browser = await Puppeteer.connect({
  apiKey: process.env.SCRAPELESS_API_KEY,
  sessionName: 'news-crawler',
  proxyCountry: 'US',
  sessionTTL: 300,
});

const page = await browser.newPage();

Step 2 — Discover article links

Load the front page and collect the links that look like articles. The right selector depends on the site, but the principle is the same everywhere: filter the anchors down to the ones that point at stories. Here it's the front page of a news source, taking the first handful of headline links:

await page.goto('https://text.npr.org/', {
  waitUntil: 'domcontentloaded',
  timeout: 40000,
});

const links = await page.evaluate(() =>
  [...document.querySelectorAll('ul li a')]
    .map((a) => ({ title: a.textContent.trim(), href: a.href }))
    .filter((l) => l.title),
);

console.log(links.length, 'links —', links[0]);
// 5 links — { title: '...', href: 'https://text.npr.org/<article-id>' }

For most outlets you tighten the filter to a URL shape — for example links whose path matches /news/articles/ or ends in a long numeric id — so navigation and section links are excluded:

const articleLinks = await page.evaluate(() =>
  [...document.querySelectorAll('a[href]')]
    .filter((a) => /\/news\/(articles\/|[a-z]+-\d{6,})/.test(a.getAttribute('href') || ''))
    .map((a) => a.href),
);

Step 3 — Fetch and extract each article

Now visit each link and pull the readable core — the h1 and the body paragraphs. A paragraph count and text length are a quick integrity check that you got a story and not a redirect or a paywall stub:

async function fetchArticle(page, url) {
  await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 40000 });
  return page.evaluate(() => ({
    title: document.querySelector('h1')?.textContent?.trim(),
    paragraphs: document.querySelectorAll('p').length,
    text: [...document.querySelectorAll('p')].map((p) => p.textContent).join('\n\n'),
  }));
}

const article = await fetchArticle(page, links[0].href);
console.log(article.title, '—', article.paragraphs, 'paragraphs');
// "U.S. and Iran announce a deal to end the war, reopen Strait of Hormuz" — 40 paragraphs

That live run returned a 40-paragraph article (~5,500 characters of body text) from the headline link — headline and full text, no navigation chrome.

Get your API key on the free plan: app.scrapeless.com

Step 4 — Assemble the crawl

Put the two stages together: discover once, then fetch each link at a controlled pace. Keep a seen set so a scheduled re-run only fetches new stories:

const seen = new Set();
const articles = [];

for (const url of articleLinks) {
  if (seen.has(url)) continue;
  seen.add(url);
  try {
    articles.push({ url, ...(await fetchArticle(page, url)) });
  } catch {
    // skip an article that won't render; the sweep continues
  }
  await new Promise((r) => setTimeout(r, 1000)); // pace the crawl
}

await browser.close();
console.log(articles.length, 'articles');

What You Get Back

Each article is a flat record — headline plus readable body:

{
  "url": "https://text.npr.org/nx-s1-5858590",
  "title": "U.S. and Iran announce a deal to end the war, reopen Strait of Hormuz",
  "paragraphs": 40,
  "text": "..."
}
// illustrative sample — title + paragraph count are a real capture; body text elided.

A few honest observations:

• Tighten the discovery filter per site. A loose anchor grab pulls in nav and tags; match the article URL shape.
• Validate the fetch. A near-zero paragraph count usually means a redirect, a paywall, or a consent wall — check before trusting the record.
• Dedupe across sweeps. Front pages re-list the same story; key on the URL.
• Pace and pin. A delay between fetches and a pinned proxyCountry keep a section sweep from tripping rate limits.

Conclusion: discover, fetch, repeat

A news crawler is two clean stages — discover the article links from a front page, then fetch and extract each story's headline and body. Keeping them separate makes the crawl pace-able, resumable, and easy to schedule. Running on Scrapeless Scraping Browser renders the JS-built front pages and supplies the residential egress that keeps a sweep from getting rate-limited. For the multi-page side of crawling, see the pagination guide; the Scraping Browser product page and docs cover the full SDK surface. Filter discovery tightly, validate each fetch, dedupe across sweeps, and pace the crawl.

Ready to Build Your AI-Powered Data Pipeline?

Join our community to claim a free plan and connect with developers building news and content pipelines: Discord · Telegram.

Sign up at app.scrapeless.com for free Scraping Browser runtime and adapt the patterns above to the outlets and sections your workflow needs. See pricing for scale.

FAQ

Q: Is crawling news sites legal?
Collecting publicly visible content is generally permissible, but copyright and each site's Terms of Service govern what you may store and republish. Crawl public pages, respect robots exclusion rules and ToS, and consult counsel for your use case.

Q: How do I separate article links from navigation?
Filter anchors to the site's article URL shape — a /news/articles/ path or a slug ending in a long id — rather than grabbing every link on the page.

Q: How do I know the article actually fetched?
Check the paragraph count and text length. A near-zero count signals a redirect, paywall, or consent wall instead of a story.

Q: Do I need a proxy?
For a sustained sweep, yes — pin proxyCountry to the region the outlet serves and pace the crawl so you don't trip rate limits.

Q: How do I run this on a schedule for only new stories?
Keep a persisted seen set of URLs; re-run discovery on your interval and fetch only the links you haven't recorded.

Q: Can I run this without an AI agent?
Yes. It's plain Puppeteer over the Scrapeless session — no agent required.