Key Takeaways

• Watir (Web Application Testing in Ruby) is a powerful tool for web automation and scraping.
• Proxies are essential in Watir to mask your IP address and prevent anti-bot systems from blocking your scraper.
• You can configure a proxy directly when initializing the Watir::Browser instance.
• Implementing a simple random proxy rotator in Ruby is necessary for making multiple requests without being banned.
• For large-scale, reliable scraping, a premium service like Scrapeless Proxy is recommended to handle complex rotation and IP health checks automatically.

How to Set Up and Rotate Proxies with Watir (Ruby)

Watir (Web Application Testing in Ruby) is an open-source family of Ruby libraries built on top of Selenium WebDriver [1], designed for automating web browsers. It is a highly effective tool for web scraping in Ruby, but like any automation tool, it can still be blocked by websites employing anti-bot measures.

In this tutorial, you will learn the essential steps to set up a proxy with Watir to avoid detection and bans, ensuring your web scraping operations run uninterrupted.

Set up a Single Proxy With Watir

To begin, you need to install the Watir gem:

gem install watir

Next, let's create a basic script. We will initialize a new Chrome browser instance in headless mode and navigate to HTTPBin [2], a service that returns the client's IP address.

scraper.rb

require 'watir'

# initialize the browser
browser = Watir::Browser.new :chrome, headless: true

# navigate to the URL
url = 'https://httpbin.io/ip'
browser.goto(url)

# get the page content
page_content = browser.text
puts page_content

# close the browser
browser.close

Running this script will reveal your machine's actual IP address, which is a poor practice for web scraping as it exposes you to immediate blocks. To mask your request, we must integrate a proxy.

Integrating the Proxy

To use a proxy, you must define the proxy settings and pass them during the browser initialization. You can find a proxy from a list of best proxy providers.

Define the proxy settings (replace the example IP and port with your own):

scraper.rb

# ...
# define proxy
proxy = {
   http: '8.219.97.248:80',
   ssl: '8.219.97.248:80'
}

# initialize the browser with the proxy settings
browser = Watir::Browser.new :chrome, headless: true, proxy: proxy
# ...

The complete code for using a single proxy looks like this:

scraper.rb

require 'watir'

# define proxy
proxy = {
   http: '8.219.97.248:80',
   ssl: '8.219.97.248:80'
}

# initialize the browser
browser = Watir::Browser.new :chrome, headless: true, proxy: proxy

# navigate to the URL
url = 'http://httpbin.io/ip'
browser.goto(url)

# get the page content
page_content = browser.text
puts page_content

# close the browser
browser.close

The output will now reflect the IP address of the proxy server, successfully masking your original IP.

Add Rotating Proxies to Watir

Using a single proxy is only a temporary solution. If you make several requests from that one IP, your activity will still be detected and blocked. Rotating proxies are necessary to distribute your requests across multiple IP addresses, making your scraper much more resilient.

Let's build a simple rotator that randomly selects a proxy from a predefined list for each browsing session.

First, define your list of proxies and configure the Selenium WebDriver logger to reduce log noise:

scraper.rb

require 'watir'
require 'logger'

# list of proxies (replace with your own list)
proxies = [
  { http: '8.219.97.248:80', ssl: '8.219.97.248:80' },
  { http: '20.235.159.154:80', ssl: '20.235.159.154:80' },
  { http: '18.188.32.159:3128', ssl: '18.188.32.159:3128' },
]

# configure Selenium WebDriver logger
logger = Selenium::WebDriver.logger
logger.ignore(:jwp_caps, :logger_info)

Next, define a function to randomly select a proxy from the list:

scraper.rb

# ...
# function to rotate proxies
def get_rotating_proxy(proxies)
  proxies.sample
end
# ...

Finally, integrate the rotation logic into your script using a begin/rescue/ensure block for robust error handling and cleanup:

scraper.rb

# ...
begin
  # initialize the browser with a randomly selected proxy
  proxy = get_rotating_proxy(proxies)
  logger.info("Using proxy: #{proxy}")
  browser = Watir::Browser.new :chrome, headless: true, proxy: proxy

  # navigate to the URL
  url = 'https://httpbin.io/ip'
  browser.goto(url)

  # get the page content
  page_content = browser.text
  puts page_content

rescue => e
  # handle error
  logger.error("An error occurred: #{e.message}")
ensure
  # close the browser
  browser.close
end

By wrapping your code in this structure, you ensure that a different proxy is used for each run, and the browser is always closed, even if an error occurs. This is a fundamental step toward building a reliable scraper.

Automated Proxy Rotation with Scrapeless Proxy

While manual rotation with a list of free proxies is a good starting point, it is not a viable solution for professional or large-scale web scraping. Free proxies are notoriously slow, unreliable, and quickly become unusable. For high-volume tasks, such as scraping Instagram follower data or running a B2B lead generation workflow, you need a premium service that handles the complexity of proxy management automatically.

Scrapeless Proxy offers a high-performance, automated solution that is perfectly suited for Watir automation. Instead of managing a list of individual proxies, you simply route all your requests through a single, authenticated Scrapeless endpoint. The service then manages a massive pool of rotating residential and ISP IPs, automatically handling rotation, retries, and IP health checks to ensure a near-perfect success rate.

This approach allows you to focus on your Watir automation logic, knowing that the underlying proxy infrastructure is robust and reliable.

Why Scrapeless is the Ideal Proxy for Watir

• Automatic Rotation: Scrapeless handles all IP rotation, eliminating the need for the manual Ruby logic shown above.
• High Success Rate: Leveraging a massive pool of 90M+ residential IPs, Scrapeless ensures your requests bypass blocks.
• Simplified Integration: Integrate with your Watir script using a single, authenticated endpoint, making your code cleaner and more robust.
• Focus on Data: You can dedicate your time to extracting data rather than debugging proxy issues.

To test the reliability and speed of the Scrapeless residential network, you can start a free trial today:

Integrating Scrapeless with Watir

When using a premium service like Scrapeless, you will need to include authentication details. This is done by passing the credentials within the proxy string.

# Replace with your actual Scrapeless credentials
PROXY_HOST = 'gate.scrapeless.com'
PROXY_PORT = 8000
PROXY_USER = 'your_username'
PROXY_PASS = 'your_password'

# Define the authenticated proxy string
auth_proxy = "#{PROXY_USER}:#{PROXY_PASS}@#{PROXY_HOST}:#{PROXY_PORT}"

# Define proxy settings
proxy = {
   http: auth_proxy,
   ssl: auth_proxy
}

# Initialize the browser with the authenticated proxy
browser = Watir::Browser.new :chrome, headless: true, proxy: proxy
# ... your Watir automation code

By using a premium provider, you gain access to reliable, high-quality proxies that handle rotation and health checks automatically, allowing your Watir scraper to run at scale without interruption. This is a key component for any successful web scraping service for data extraction.

Conclusion

Setting up a proxy with Watir is a straightforward process that is essential for bypassing anti-bot measures. While manual rotation is possible, the most robust and scalable solution for serious web scraping in Ruby is to leverage an automated, premium proxy service like Scrapeless Proxy. This allows your Watir scripts to benefit from a vast pool of rotating, high-quality IPs, ensuring a high success rate and allowing you to focus on the core task of data extraction.

Frequently Asked Questions (FAQ)

What is Watir?

Watir, which stands for Web Application Testing in Ruby, is a family of open-source Ruby libraries used for automating web browsers. It allows you to write scripts that interact with web pages in the same way a user would, making it popular for both testing and web scraping.

Why do I need a proxy for Watir scraping?

You need a proxy to mask your computer's real IP address. When a Watir script sends a high volume of requests, the target website's anti-bot system will detect the unusual traffic pattern from a single IP and block it. A proxy, especially a rotating one, prevents this by making the requests appear to come from many different users.

Can I use a free proxy with Watir?

You can, but it is strongly discouraged for any serious project. Free proxies are typically slow, unreliable, have limited uptime, and are often already blacklisted by major websites. They are only suitable for basic testing or learning purposes. For production, you should use a premium service like Scrapeless.

Does Watir support proxy authentication?

Yes, Watir supports proxy authentication. As shown in the integration example, you can include the username and password directly in the proxy string when defining the proxy settings for the Watir::Browser instance.

Is Watir better than Selenium for Ruby?

Watir is built on top of Selenium WebDriver and is often considered more "Ruby-friendly" due to its clean, expressive API. For users already working in the Ruby ecosystem, Watir provides a more idiomatic and readable way to interact with the browser compared to using the raw Selenium bindings.

References

[1] Selenium WebDriver Documentation
[2] HTTPBin Service
[3] The Official Ruby Programming Language Website
[4] Watir Official Website
[5] W3C HTTP Proxy Specification