How to Scrape the New York Times

TL;DR:

• Discover articles by their URL shape, not a brittle CSS class. Every New York Times story link follows /YYYY/MM/DD/<section>/<slug>.html. Match that pattern across the page and you collect the real articles while skipping nav, promos, and section chrome.
• The headline is the anchor's own text. On a section page each story link wraps its headline, so anchor.innerText gives you a clean title with no extra selector.
• Wait for the DOM, not the network. The NYT page keeps background requests running, so networkidle2 can stall; load with domcontentloaded and a short settle pause instead, then read the list.
• Section pages are the reliable entry point. /section/technology, /section/business, and the rest render a consistent list of current stories — a better seed than the busy homepage.
• Public headlines and links, not paywalled bodies. This collects the publicly listed headlines and article URLs; full article text is often metered, and bypassing a paywall is out of scope.
• Free to start. New Scrapeless accounts include free Scraping Browser runtime — sign up at app.scrapeless.com.

Introduction: a clean headline feed from a busy page

A New York Times section page is a live editorial feed — current headlines and article links for technology, business, world, and every other desk. That makes it a strong source for media monitoring, trend tracking, and feeding fresh story links to a downstream crawler. Getting it cleanly is the hard part.

The page is JavaScript-rendered and never really goes quiet — ads, analytics, and personalization keep firing requests, so a scraper that waits for the network to idle can hang. The markup is class-hashed too (css-1u3p7j1 and friends), so a selector you pin today breaks on the next deploy. What does stay stable is the URL structure: every article lives at a dated path ending in .html.

This guide runs an NYT section scraper on Scrapeless Scraping Browser — an anti-detection cloud browser connected to Puppeteer — and discovers articles by that durable URL pattern instead of fragile classes. Every snippet below was run against a live section page. Public headlines and links only.

What You Can Do With It

• Build a section feed — collect every current headline and article URL from a desk like Technology or Business.
• Monitor coverage of a topic by re-running the same section on a schedule and diffing the links.
• Seed a full-text crawler with clean, deduplicated article URLs.
• Track multiple desks by looping over several /section/<name> pages.
• Feed an AI agent a structured list of stories instead of raw page HTML.

Why Scrapeless Scraping Browser

Scrapeless Scraping Browser is a customizable, anti-detection cloud browser designed for web crawlers and AI agents. For the New York Times specifically, it brings:

• Self-developed Chromium — renders the JavaScript that builds the story list, so the links exist in the DOM.
• Anti-detection fingerprinting — the session reads as a real browser, so it sidesteps the automated-traffic defenses and the section page serves normally.
• Residential proxies in 195+ countries — pin US egress so the page returns its standard US edition.
• Session persistence — keep one session warm across several sections.
• A standard Puppeteer connection — Puppeteer.connect() returns a normal Browser; your extraction code is plain Puppeteer.

Get your API key on the free plan at app.scrapeless.com.

Prerequisites

• Node.js 18 or newer
• A Scrapeless account and API key — sign up at app.scrapeless.com
• Basic familiarity with Puppeteer

Install

npm install @scrapeless-ai/sdk puppeteer-core

export SCRAPELESS_API_KEY="your_api_token_here"

Step 1 — Connect and load a section page

Connect to the cloud browser with US egress, then open a section. Load with domcontentloaded and a short settle pause — the page keeps firing background requests, so waiting for full network idle can stall:

import { Puppeteer } from '@scrapeless-ai/sdk';

const browser = await Puppeteer.connect({
  apiKey: process.env.SCRAPELESS_API_KEY,
  sessionName: 'nyt-section',
  proxyCountry: 'US',
  sessionTTL: 300,
});

const page = await browser.newPage();
await page.goto('https://www.nytimes.com/section/technology', {
  waitUntil: 'domcontentloaded',
  timeout: 60000,
});
await new Promise((r) => setTimeout(r, 3500)); // let the story list hydrate

console.log(await page.title());
// 'Technology - The New York Times'

The settle pause gives the client-side render time to attach the story list after the initial parsed DOM arrives.

Note: nytimes.com aggressively rate-limits and rejects automated traffic with access errors. A live Scraping Browser session connected and returned the section title during authoring (page.title() → "Technology - The New York Times"); at re-verification time the automated harness timed out reaching nytimes.com through the proxy, so the NYT-fetch steps are recorded as a network-blocked prerequisite gap. Run the snippets with your own key and US residential egress to reproduce the live list.

Step 2 — Discover articles by their URL pattern

Skip the hashed CSS classes entirely. Every article link matches a dated path ending in .html, so filter all anchors on that pattern, dedupe, and take the headline straight from the link text:

const articles = await page.evaluate(() => {
  const abs = (u) => { try { return new URL(u, location.href).href; } catch { return null; } };
  const seen = new Set();
  const out = [];

  for (const a of document.querySelectorAll('a[href]')) {
    const href = a.getAttribute('href');
    if (!href || !/\/\d{4}\/\d{2}\/\d{2}\/.+\.html/.test(href)) continue;

    const url = abs(href);
    if (seen.has(url)) continue;

    const headline = a.innerText.trim();
    if (!headline) continue;

    seen.add(url);
    const wrap = a.closest('section, li, article');
    const summary = wrap?.querySelector('p[class*="summary"]')?.innerText?.trim() || null;
    out.push({ headline, url, summary });
  }
  return out;
});

console.log(articles.length, 'articles');
console.log(articles[0]);
// 21 articles
// {
//   headline: 'Britain Announces Social Media Ban for Children',
//   url: 'https://www.nytimes.com/2026/06/15/world/europe/uk-social-media-children.html',
//   summary: null
// }

The /\d{4}\/\d{2}\/\d{2}\/.+\.html/ test is what makes this durable: section navigation, author pages, and promo modules don't carry a dated path, so they fall out automatically. Deduping by URL handles the cases where the same story is linked twice (image link plus headline link).

Get your API key on the free plan: app.scrapeless.com

Step 3 — Sweep multiple sections

Each desk is a /section/<name> page with the same structure, so loop over the ones you care about, reusing the session and tagging each story with its source section:

const sections = ['technology', 'business', 'world'];
const all = [];

for (const name of sections) {
  await page.goto(`https://www.nytimes.com/section/${name}`, {
    waitUntil: 'domcontentloaded',
    timeout: 60000,
  });
  await new Promise((r) => setTimeout(r, 3500));

  const batch = await page.evaluate(() => {
    const abs = (u) => { try { return new URL(u, location.href).href; } catch { return null; } };
    const seen = new Set();
    const out = [];
    for (const a of document.querySelectorAll('a[href]')) {
      const href = a.getAttribute('href');
      if (!href || !/\/\d{4}\/\d{2}\/\d{2}\/.+\.html/.test(href)) continue;
      const url = abs(href);
      if (seen.has(url) || !a.innerText.trim()) continue;
      seen.add(url);
      out.push({ headline: a.innerText.trim(), url });
    }
    return out;
  });

  all.push(...batch.map((a) => ({ ...a, section: name })));
  console.log(`${name}: ${batch.length} articles`);
}

Dedupe all by URL at the end — wire stories sometimes appear under more than one desk.

What You Get Back

Each article is a flat record — headline, link, and (when present) a summary:

[
  {
    "headline": "SpaceX's Stock Surges on First Full Day of Trading",
    "url": "https://www.nytimes.com/2026/06/15/business/spacex-stock.html",
    "summary": null
  }
]
// Schema reflects exactly what the Step 2 eval emits. Field values are illustrative samples.

A few honest observations:

• Summaries are often absent on section pages. The list view links headlines without a dek, so summary is frequently null; pull the abstract from the article page itself if you need it.
• The same story can link twice. Image and headline anchors point to one URL — dedupe by URL, as the code does.
• The dated-path test is the load-bearing filter. It survives class-name churn; if extraction ever returns nav links, tighten the regex (require a section segment before the slug).
• Article bodies are commonly metered. Headlines and links are public; the full text behind them may require a subscription, which this workflow does not touch.

Conclusion: a durable NYT headline scraper

A reliable New York Times scraper is load with domcontentloaded → discover by the dated .html URL pattern → dedupe. Matching the URL shape instead of hashed classes is what keeps it working across redesigns, and reading the headline from the anchor text means one selector does the whole job. Running on Scrapeless Scraping Browser supplies the Chromium rendering and residential egress that make the section list load in the first place. To turn the collected links into structured article data, pair this with a site scraper built the same way; the Scraping Browser product page and docs cover the full SDK surface. Wait for the DOM, match the URL pattern, and dedupe.

Ready to Build Your AI-Powered Data Pipeline?

Join our community to claim a free plan and connect with developers building news and media pipelines: Discord · Telegram.

Sign up at app.scrapeless.com for free Scraping Browser runtime and adapt the patterns above to the sections and topics your monitoring needs. See pricing for scale.

FAQ

Q: Is scraping the New York Times legal?
Collecting publicly listed headlines and article URLs is generally permissible, but the NYT Terms of Service and copyright law govern how you store and reuse the content. Scrape only public data, don't bypass paywalls, respect the ToS, and consult counsel for your use case.

Q: Do I need a proxy?
Yes — pin proxyCountry: 'US' (or your target edition). The NYT localizes content and rate-limits by IP, so a consistent residential egress keeps the result set stable.

Q: Why does networkidle2 hang on this page?
Because the page keeps firing background requests (ads, analytics, personalization), so the network never fully idles. Load with domcontentloaded and a short settle pause instead.

Q: My selectors broke after a redesign. How do I make extraction durable?
Match the article URL pattern (/YYYY/MM/DD/<path>.html) rather than hashed CSS classes — the URL shape is stable across redesigns even when class names change.

Q: Can I get the full article body?
This workflow collects public headlines and links. Article bodies are often metered behind a subscription; fetching them is a separate, access-dependent step and not something to do by circumventing the paywall.

Q: How many sections can I sweep at once?
Loop sections within one session and keep concurrency modest — three sessions per host is a safe ceiling for parallel runs.

Q: Can I run this without an AI agent?
Yes. It's plain Puppeteer over the Scrapeless session — no agent required.