Ruby Web Scraping: A Practical Guide

TL;DR:

• Ruby's fetch-and-parse story is two lines of setup. Net::HTTP from the standard library plus Nokogiri covers server-rendered pages completely.
• Nokogiri takes CSS selectors, so most examples port directly. doc.css("div.quote") behaves the way the same selector behaves in a browser console.
• The Scrapeless Universal Scraping API answers with a JSON envelope. The markup arrives inside data, so you parse JSON first and HTML second.
• Ferrum cannot reach a TLS browser endpoint out of the box, and the reason is one missing line. It builds its CDP socket without setting hostname, so no SNI is sent and an SNI-dependent host refuses the handshake. Confirmed in isolation: identical socket, no SNI fails, SNI succeeds against a certificate for scrapeless.com.
• A fifteen-line patch makes it work. With SNI supplied, the same script returned title: Quotes to Scrape and quotes on page: 10.
• Start free: the Scrapeless dashboard issues a key that works with every example below.

What You Need

Everything below ran on Ruby 3.2.3 against quotes.toscrape.com, a site published for scraping practice.

gem install nokogiri ferrum --no-document
# nokogiri 1.19.4 · ferrum 0.17.2

Net::HTTP, URI and JSON are standard library, so the only real dependencies are the parser and, later, the browser driver.

One naming quirk worth knowing before you write a version banner: Nokogiri exposes Nokogiri::VERSION, but Ferrum defines no Ferrum::VERSION constant. Referencing it raises NameError: uninitialized constant Ferrum::VERSION. Read the version from Gem.loaded_specs["ferrum"].version instead.

Fetching and Parsing

require "net/http"
require "uri"
require "nokogiri"

uri = URI("https://quotes.toscrape.com/")
res = Net::HTTP.get_response(uri)
puts "http=#{res.code} bytes=#{res.body.bytesize}"

doc    = Nokogiri::HTML(res.body)
quotes = doc.css("div.quote")
puts "quotes=#{quotes.size}"
puts "first_author=#{quotes.first.at_css('small.author').text}"
puts "first_text=#{quotes.first.at_css('span.text').text[0, 40]}"

http=200 bytes=11064
quotes=10
first_author=Albert Einstein
first_text=“The world as we have created it is a pr

Two habits pay off immediately. css returns a node set and at_css returns the first match or nil, so reach for at_css whenever you want one element — it is the difference between nil and an empty set when a field is missing, and nil fails loudly at the point of the mistake.

Scope child queries to the row. quotes.first.at_css('span.text') searches inside that node, whereas calling doc.at_css in a loop returns the first quote's text on every iteration. That produces a plausible-looking dataset where every row carries the same value, which is far worse than a crash.

Nokogiri's documentation covers the XPath equivalents if you prefer them; doc.xpath("//div[@class='quote']") is the same query in the other dialect.

Where Plain Ruby Stops

The script works because the target renders server-side and does not screen its callers. Two situations end that: content that only exists after JavaScript runs, and sites that treat a bare HTTP client as a bot. Neither is a Ruby limitation — no HTTP client in any language solves either.

From here the escalations are different tools rather than better versions of each other, and the plain path stays the fastest and cheapest where it works. If you need background on what a browser adds, the browser automation explainer covers the general shape.

Escalation One: The Universal Scraping API

This keeps your code an ordinary HTTP client and moves the difficulty server-side.

require "net/http"
require "uri"
require "json"
require "nokogiri"

key = ENV.fetch("SCRAPELESS_API_KEY")
api = URI("https://api.scrapeless.com/api/v2/unlocker/request")

req = Net::HTTP::Post.new(api, "Content-Type" => "application/json", "x-api-token" => key)
req.body = JSON.dump(
  actor: "unlocker.webunlocker",
  input: { url: "https://quotes.toscrape.com/", js_render: false }
)

resp = Net::HTTP.start(api.hostname, api.port, use_ssl: true, read_timeout: 90) { |h| h.request(req) }
body = JSON.parse(resp.body)

puts "http=#{resp.code} envelope_keys=#{body.keys.join(',')}"
puts "quotes=#{Nokogiri::HTML(body['data']).css('div.quote').size}"

http=200 envelope_keys=code,data
quotes=10

The envelope is the part to internalise. resp.body is JSON; the markup lives at body["data"]. Handing resp.body straight to Nokogiri::HTML produces a document with no matching nodes and raises nothing — selectors quietly return zero rows. Parse the JSON, take data, then parse the HTML, and keep that unwrap in one method rather than scattering JSON.parse(...)["data"] across the codebase.

Note Net::HTTP.start(..., use_ssl: true) rather than the shorter Net::HTTP.post. Omitting use_ssl against an https URI is a common Ruby stumble that surfaces as a confusing connection reset rather than a clear error.

Escalation Two: A Real Browser, and a Gem Bug

Ferrum is Ruby's Chrome DevTools Protocol driver, and it accepts a ws_url: for attaching to a browser you did not launch. Pointed at a TLS endpoint, it fails:

OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 peeraddr=172.67.74.62:443
state=error: sslv3 alert handshake failure

That message names the wrong layer. It is not a certificate problem, a cipher problem, or a proxy problem.

The Actual Cause

Ferrum 0.17.2 builds its CDP socket in lib/ferrum/client/web_socket.rb like this:

tcp = TCPSocket.new(uri.host, port)
ssl_context = OpenSSL::SSL::SSLContext.new
@sock = OpenSSL::SSL::SSLSocket.new(tcp, ssl_context)
@sock.sync_close = true
@sock.connect

OpenSSL::SSL::SSLSocket#hostname is never assigned, so Ruby sends no Server Name Indication extension. A host that serves many certificates from one address cannot choose one, and answers with a handshake failure.

Twenty lines of plain Ruby isolate it — identical sockets, one line different:

require "socket"
require "openssl"

HOST = "browser.scrapeless.com"

def attempt(sni)
  tcp  = TCPSocket.new(HOST, 443)
  sock = OpenSSL::SSL::SSLSocket.new(tcp, OpenSSL::SSL::SSLContext.new)
  sock.hostname = HOST if sni      # the line ferrum omits
  sock.sync_close = true
  sock.connect
  cn = sock.peer_cert.subject.to_a.find { |a| a[0] == "CN" }
  result = "OK  cert_cn=#{cn && cn[1]}"
  sock.close
  result
rescue => e
  "FAIL #{e.class}: #{e.message.to_s[0, 70]}"
end

puts "without SNI: #{attempt(false)}"
puts "with SNI:    #{attempt(true)}"

without SNI: FAIL OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 peeraddr=172.67.74.62:443
with SNI:    OK  cert_cn=scrapeless.com

That is the whole bug. Any Ruby client omitting hostname= hits it against any SNI-dependent endpoint, which today is most of them.

A Red Herring Worth Naming

There is a second suspicious behaviour in Ferrum, and it is not the cause. Ferrum::Browser::Process.parse_json_version runs URI.join(url, "/json/version") against your ws_url, which replaces the path and discards the query string. A wss:// URL whose path is /api/v2/browser and whose query carries your token is rewritten to the host root plus /json/version, dropping the credential entirely. That rewritten address answers 404 page not found here.

It looks fatal and is not: the method rescues JSON::ParserError, and a 404 body is not valid JSON, so the failure is swallowed. Fixing SNI alone is sufficient — the patched run below still 404s on that probe and works anyway. Worth knowing so you do not spend an afternoon on it, as this article's first diagnosis did.

Making It Work

The CDP socket is typically the only TLS socket a scraper of this shape opens, so supplying the name at construction is enough:

require "ferrum"
require "openssl"

module SNIPatch
  def connect
    self.hostname = @ferrum_sni if @ferrum_sni && respond_to?(:hostname=)
    super
  end
end

class OpenSSL::SSL::SSLSocket
  prepend SNIPatch

  def ferrum_sni=(host)
    @ferrum_sni = host
  end
end

module SSLSocketFactoryPatch
  def new(io, ctx = nil)
    sock = super
    sock.ferrum_sni = Thread.current[:ferrum_sni_host]
    sock
  end
end

class << OpenSSL::SSL::SSLSocket
  prepend SSLSocketFactoryPatch
end

key = ENV.fetch("SCRAPELESS_API_KEY")
Thread.current[:ferrum_sni_host] = "browser.scrapeless.com"

browser = Ferrum::Browser.new(
  ws_url: "wss://browser.scrapeless.com/api/v2/browser?token=#{key}",
  timeout: 60,
  process_timeout: 60
)
browser.go_to("https://quotes.toscrape.com/")
puts "title: #{browser.evaluate('document.title')}"
puts "quotes on page: #{browser.evaluate('document.querySelectorAll(".quote").length')}"
browser.quit

title: Quotes to Scrape
quotes on page: 10

Two notes on scope. The patch is global to OpenSSL::SSL::SSLSocket, which is acceptable in a single-purpose scraper and not something to load into a Rails application that opens other TLS sockets — there, confine it to the process that drives the browser. And the upstream fix is a single line in the gem, so check whether a release past 0.17.2 has landed it before carrying a patch of your own.

Choosing Between the Three

Work down the list rather than up. A page that appears to need a browser very often embeds its data in a <script> tag, and doc.at_css("script#__NEXT_DATA__") with JSON.parse beats a browser session on every axis.

Conclusion

Ruby handles scraping well, and the standard library carries more of it than people expect. Net::HTTP and Nokogiri cover server-rendered pages, with at_css versus css and row-scoped child queries as the two details that separate correct extraction from a dataset that looks fine and is not.

The browser path is where Ruby currently costs more than its neighbours, and the reason is narrow rather than fundamental: one unset attribute in one gem, producing an error message that points at TLS instead of at SNI. The isolation script above answers it in twenty lines, and the patch is small enough to carry until upstream ships the one-line fix.

Ready to Scrape With Ruby?

Create a key on the Scrapeless dashboard and run the Net::HTTP example against a page you already collect. If it returns what you expect, you are finished — one gem, no browser. The Universal Scraping API covers the pages where it does not, and current rates are on the pricing page.

FAQ

Q: Nokogiri or an alternative parser?

Nokogiri remains the default for HTML. It takes both CSS selectors and XPath, is well documented, and handles malformed markup. Lighter pure-Ruby parsers exist and are worth considering only if native extensions are a problem in your deployment.

Q: Why does my Ferrum connection fail with an SSL handshake error?

Because Ferrum 0.17.2 does not set hostname on its OpenSSL::SSL::SSLSocket, so no SNI is sent and an SNI-dependent endpoint cannot select a certificate. Reproduce it in isolation with the twenty-line script above, then apply the patch or wait for the upstream fix.

Q: Why does my parsing return nothing when the API call clearly succeeded?

You are parsing the envelope. The response is JSON with the markup inside data, so Nokogiri receives a JSON string, matches nothing, and raises nothing. Parse the JSON first and hand body["data"] to Nokogiri.

Q: Does Ferrum::VERSION exist?

No. It raises NameError: uninitialized constant Ferrum::VERSION. Use Gem.loaded_specs["ferrum"].version when you want to log the version.

Q: Is Ruby a reasonable choice for large scraping jobs?

For fetch-and-parse work, yes — Nokogiri is a fast native parser and threads handle IO-bound concurrency well. The weaker area is browser automation, where the surrounding tooling is thinner than Python's or Node's, as the SNI issue above illustrates. A common split is Ruby for the HTTP path and a hosted browser for the pages that need one.