🥳Join the Scrapeless Community and Claim Your Free Trial to Access Our Powerful Web Scraping Toolkit!
Back to Blog

How to Scrape Google Trends Data With Python?

Alex Johnson
Alex Johnson

Senior Web Scraping Engineer

25-Dec-2024

What Is Google Trends?

Google Trends is a free online tool provided by Google that analyzes the popularity of specific keywords or search terms in the Google search engine over time.

It presents data in the form of charts to help users understand the search popularity of a certain topic or keyword, and identifies patterns such as seasonal fluctuations, emerging trends, or declining interest. Google Trends not only supports global data analysis, but also can be refined to specific regions and provide recommendations for related search terms and topics.

Google Trends is widely used in market research, content planning, SEO optimization, and user behavior analysis, helping users make more informed decisions based on data.

For example: In this article, let's scrape the Google search trends of 'DOGE' from the previous month.

Google search trends of DOGE

Step 1: Prerequisites

Install Python

On Windows

Using the Official Python Installer

  1. Download Python Installer:

    • Go to the official Python website.
    • The website should automatically suggest the latest version for Windows. Click the Download Python button to download the installer.
  2. Run the Installer:

    • Open the downloaded .exe file to start the installation process.
  3. Customize Installation (optional):

    • Make sure to check the box that says "Add Python to PATH" at the beginning of the installation window. This makes Python accessible from the command line (cmd or PowerShell).
    • You can also click "Customize installation" to choose additional features like pip, IDLE, or documentation.
  4. Install Python:

    • Click Install Now to install Python with the default settings.
    • After installation, you can verify it by opening Command Prompt (cmd) and typing:
      bash Copy
      python --version
  5. Installing pip (if needed):

    • Pip, the Python package manager, is installed by default with modern versions of Python. You can check if pip is installed by typing:
      bash Copy
      pip --version

You can also install Python directly from the Windows Store (available on Windows 10/11). Simply search for "Python" in the Microsoft Store app and choose the version you need.

On macOS

Method 1. Using Homebrew (Recommended)

  1. Install Homebrew (if not already installed):

    • Open the Terminal app.
    • Paste the following command to install Homebrew (package manager for macOS):
      bash Copy
      /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
  2. Install Python with Homebrew:

    • Once Homebrew is installed, you can install Python with this command:
      bash Copy
      brew install python
  3. Verify Installation:

    • After installation, you can verify Python and pip versions with the following commands:
      bash Copy
      python3 --version
      pip3 --version

Method 2. Using the Official Python Installer

  1. Download the macOS Installer:

  2. Run the Installer:

    • Open the .pkg file to start the installation process and follow the instructions.
  3. Verify Installation:

    • After installation, open Terminal and check the Python version:
      bash Copy
      python3 --version
      pip3 --version

On Linux

For Debian/Ubuntu-based Distributions

  1. Update Package List:

    • Open a terminal and run the following command to update the package list:
      bash Copy
      sudo apt update
  2. Install Python:

    • To install Python 3 (usually the latest version of Python 3.x), run:
      bash Copy
      sudo apt install python3
  3. Install pip (if not installed):

    • If pip is not already installed, you can install it with:
      bash Copy
      sudo apt install python3-pip
  4. Verify Installation:

    • To check the installed Python version:
      bash Copy
      python3 --version
      pip3 --version

For Red Hat/Fedora-based Distributions

  1. Install Python 3:

    • Open a terminal and run:
      bash Copy
      sudo dnf install python3
  2. Install pip (if necessary):

    • If pip is not installed by default, you can install it with:
      bash Copy
      sudo dnf install python3-pip
  3. Verify Installation:

    • To check the installed Python version:
      bash Copy
      python3 --version
      pip3 --version

For Arch Linux and Arch-based Distros

  1. Install Python 3:

    • Run the following command:
      bash Copy
      sudo pacman -S python
  2. Install pip:

    • Pip should be installed with Python, but if not, you can install it with:
      bash Copy
      sudo pacman -S python-pip
  3. Verify Installation:

    • To check the Python and pip versions:
      bash Copy
      python --version
      pip --version

Using Python via Anaconda (Cross-platform)

Anaconda is a popular distribution for scientific computing and comes with Python, libraries, and the conda package manager.

  1. Download Anaconda:

  2. Install Anaconda:

    • Follow the installation instructions based on your operating system. Anaconda provides a graphical installer for both Windows and macOS, as well as command-line installers for all platforms.
  3. Verify Installation:

    • After installation, open a terminal (or Anaconda Prompt on Windows) and check if Python is working:

      bash Copy
      python --version
    • You can also verify conda (the package manager for Anaconda):

      bash Copy
      conda --version

Managing Python Versions (optional)

If you need to manage multiple Python versions on the same machine, you can use version managers:

  • pyenv: A popular Python version manager that works on Linux and macOS.

    • Install via Homebrew or GitHub (for Linux and macOS).
    • On Windows, you can use pyenv-win.
    bash Copy
    pyenv install 3.9.0
    pyenv global 3.9.0

Since we have not yet developed a third-party library for use, you only need to install requests to experience the scrapeless API service

Shell Copy
pip install requests

Step 2: Configure the code fields that need

Write the code

Next, we need to know how to obtain the data we need through configuration:

  • Keyword: In this example, our keyword is 'DOGE' (we also support the collection of multiple keyword comparison data)
  • Data configuration:
    • Country: Query country, the default is 'Worldwide'
    • Time: Time period
    • Category: Type
    • Property: Source

Step 3: Extracting data

Now, let's get the target data using Python code:

Python Copy
import json
import requests


class Payload:
    def __init__(self, actor, input_data, proxy):
        self.actor = actor
        self.input = input_data
        self.proxy = proxy


def send_request(data_type, search_term):
    host = "api.scrapeless.com"
    url = f"https://{host}/api/v1/scraper/request"
    token = "scrapeless-api-key"  # TODO:use your api key

    headers = {"x-api-token": token}

    input_data = {
        "q": search_term,
        "date": "today 1-m",
        "data_type": data_type,
        "hl": "en-sg",
        "tz": "-480",
        "geo": "",
        "cat": "",
        "property": "",
    }

    proxy = {
        "country": "ANY",
    }

    payload = Payload("scraper.google.trends", input_data, proxy)

    json_payload = json.dumps(payload.__dict__)

    response = requests.post(url, headers=headers, data=json_payload)

    if response.status_code != 200:
        print("Error:", response.status_code, response.text)
        return

    print("body", response.text)


if __name__ == "__main__":
    send_request(data_type="interest_over_time", search_term="DOGE")
  • Output:
JSON Copy
{"interest_over_time":{"averages":[],"timelineData":[{"formattedAxisTime":"24 Nov","formattedTime":"24 Nov 2024","formattedValue":["85"],"hasData":[true],"time":"1732406400","value":[85]},{"formattedAxisTime":"25 Nov","formattedTime":"25 Nov 2024","formattedValue":["89"],"hasData":[true],"time":"1732492800","value":[89]},{"formattedAxisTime":"26 Nov","formattedTime":"26 Nov 2024","formattedValue":["68"],"hasData":[true],"time":"1732579200","value":[68]},{"formattedAxisTime":"27 Nov","formattedTime":"27 Nov 2024","formattedValue":["60"],"hasData":[true],"time":"1732665600","value":[60]},{"formattedAxisTime":"28 Nov","formattedTime":"28 Nov 2024","formattedValue":["49"],"hasData":[true],"time":"1732752000","value":[49]},{"formattedAxisTime":"29 Nov","formattedTime":"29 Nov 2024","formattedValue":["55"],"hasData":[true],"time":"1732838400","value":[55]},{"formattedAxisTime":"30 Nov","formattedTime":"30 Nov 2024","formattedValue":["54"],"hasData":[true],"time":"1732924800","value":[54]},{"formattedAxisTime":"1 Dec","formattedTime":"1 Dec 2024","formattedValue":["55"],"hasData":[true],"time":"1733011200","value":[55]},{"formattedAxisTime":"2 Dec","formattedTime":"2 Dec 2024","formattedValue":["64"],"hasData":[true],"time":"1733097600","value":[64]},{"formattedAxisTime":"3 Dec","formattedTime":"3 Dec 2024","formattedValue":["57"],"hasData":[true],"time":"1733184000","value":[57]},{"formattedAxisTime":"4 Dec","formattedTime":"4 Dec 2024","formattedValue":["61"],"hasData":[true],"time":"1733270400","value":[61]},{"formattedAxisTime":"5 Dec","formattedTime":"5 Dec 2024","formattedValue":["100"],"hasData":[true],"time":"1733356800","value":[100]},{"formattedAxisTime":"6 Dec","formattedTime":"6 Dec 2024","formattedValue":["84"],"hasData":[true],"time":"1733443200","value":[84]},{"formattedAxisTime":"7 Dec","formattedTime":"7 Dec 2024","formattedValue":["79"],"hasData":[true],"time":"1733529600","value":[79]},{"formattedAxisTime":"8 Dec","formattedTime":"8 Dec 2024","formattedValue":["72"],"hasData":[true],"time":"1733616000","value":[72]},{"formattedAxisTime":"9 Dec","formattedTime":"9 Dec 2024","formattedValue":["64"],"hasData":[true],"time":"1733702400","value":[64]},{"formattedAxisTime":"10 Dec","formattedTime":"10 Dec 2024","formattedValue":["64"],"hasData":[true],"time":"1733788800","value":[64]},{"formattedAxisTime":"11 Dec","formattedTime":"11 Dec 2024","formattedValue":["63"],"hasData":[true],"time":"1733875200","value":[63]},{"formattedAxisTime":"12 Dec","formattedTime":"12 Dec 2024","formattedValue":["59"],"hasData":[true],"time":"1733961600","value":[59]},{"formattedAxisTime":"13 Dec","formattedTime":"13 Dec 2024","formattedValue":["54"],"hasData":[true],"time":"1734048000","value":[54]},{"formattedAxisTime":"14 Dec","formattedTime":"14 Dec 2024","formattedValue":["48"],"hasData":[true],"time":"1734134400","value":[48]},{"formattedAxisTime":"15 Dec","formattedTime":"15 Dec 2024","formattedValue":["43"],"hasData":[true],"time":"1734220800","value":[43]},{"formattedAxisTime":"16 Dec","formattedTime":"16 Dec 2024","formattedValue":["48"],"hasData":[true],"time":"1734307200","value":[48]},{"formattedAxisTime":"17 Dec","formattedTime":"17 Dec 2024","formattedValue":["55"],"hasData":[true],"time":"1734393600","value":[55]},{"formattedAxisTime":"18 Dec","formattedTime":"18 Dec 2024","formattedValue":["52"],"hasData":[true],"time":"1734480000","value":[52]},{"formattedAxisTime":"19 Dec","formattedTime":"19 Dec 2024","formattedValue":["63"],"hasData":[true],"time":"1734566400","value":[63]},{"formattedAxisTime":"20 Dec","formattedTime":"20 Dec 2024","formattedValue":["64"],"hasData":[true],"time":"1734652800","value":[64]},{"formattedAxisTime":"21 Dec","formattedTime":"21 Dec 2024","formattedValue":["47"],"hasData":[true],"time":"1734739200","value":[47]},{"formattedAxisTime":"22 Dec","formattedTime":"22 Dec 2024","formattedValue":["44"],"hasData":[true],"time":"1734825600","value":[44]},{"formattedAxisTime":"23 Dec","formattedTime":"23 Dec 2024","formattedValue":["44"],"hasData":[true],"time":"1734912000","value":[44]},{"formattedAxisTime":"24 Dec","formattedTime":"24 Dec 2024","formattedValue":["46"],"hasData":[true],"isPartial":true,"time":"1734998400","value":[46]}]}}

Step 4: Optimize code

  • Configure multiple countries
Python Copy
country_map = {
        "Worldwide": "ANY",
        "Afghanistan":"AF",
        "Åland Islands":"AX",
        "Albania":"AL",
        #...
    }
  • Configure multiple time periods
Python Copy
time_map = {
        "Past hour":"now 1-H",
        "Past 4 hours":"now 4-H",
        "Past 7 days":"now 7-d",
        "Past 30 days":"today 1-m",
        # ...
    }
  • Configure multiple categories
Python Copy
category_map = {
        "All categories": 0,
        "Arts & Entertainment": 3,
        "Autos & Vehicles": 47,
        # ...
    }
  • Configure multiple sources
Python Copy
property_map = {
        "Web Search":"",
        "Image Search":"images",
        "Google Shopping":"froogle",
         # ...
    }
  • Improved code:
Python Copy
import json
import requests

country_map = {
    "Worldwide": "",
    "Afghanistan": "AF",
    "Åland Islands": "AX",
    "Albania": "AL",
    # ...
}
time_map = {
    "Past hour": "now 1-H",
    "Past 4 hours": "now 4-H",
    "Past 7 days": "now 7-d",
    "Past 30 days": "today 1-m",
    # ...
}
category_map = {
    "All categories": "",
    "Arts & Entertainment": "3",
    "Autos & Vehicles": "47",
    # ...
}
property_map = {
    "Web Search": "",
    "Image Search": "images",
    "Google Shopping": "froogle",
    # ...
}


class Payload:
    def __init__(self, actor, input_data, proxy):
        self.actor = actor
        self.input = input_data
        self.proxy = proxy


def send_request(data_type, search_term, country, time, category, property):
    host = "api.scrapeless.com"
    url = f"https://{host}/api/v1/scraper/request"
    token = "scrapeless-api-key"  # TODO:use your api key

    headers = {"x-api-token": token}

    input_data = {
        "q": search_term,  # search term
        "geo": country,
        "date": time,
        "cat": category,
        "property": property,
        "hl": "en-sg",
        "tz": "-480",
        "data_type": data_type
    }

    proxy = {
        "country": "ANY",
    }

    payload = Payload("scraper.google.trends", input_data, proxy)

    json_payload = json.dumps(payload.__dict__)

    response = requests.post(url, headers=headers, data=json_payload, verify=False)

    if response.status_code != 200:
        print("Error:", response.status_code, response.text)
        return

    print("body", response.text)


if __name__ == "__main__":
    # one search_term
    send_request(
        data_type="interest_over_time",
        search_term="DOGE",
        country=country_map["Worldwide"],
        time=time_map["Past 30 days"],
        category=category_map["Arts & Entertainment"],
        property=property_map["Web Search"],
    )
    # two search_term
    send_request(
        data_type="interest_over_time",
        search_term="DOGE,python",
        country=country_map["Worldwide"],
        time=time_map["Past 30 days"],
        category=category_map["Arts & Entertainment"],
        property=property_map["Web Search"],
    )

Problems in the crawling process

  • We need to make judgments on some network errors to prevent errors from causing shutdowns;
  • Adding a certain retry mechanism can prevent interruptions in the crawling process from causing duplicate/invalid data acquisition.
  • Step 1. Log in to Scrapeless
  • Step 2. Click the "Scraping API"
Scraping API
  • Step 3. Find our "Google Trends" Panel and enter it:
Google Trends
  • Step 4. Configure your data in the left operation panel:
Configure your data
  • Step 5. Click the "Start Scraping" button and then you can get the result:
Start Scraping

Besides, you can also refer to our sample codes.

Scrapeless is an innovative solution designed to simplify the process of extracting data from websites. Our API is designed to navigate the most complex web environments and effectively manage dynamic content and JavaScript rendering.

Why Scrapeless works well to scrape Google Trends?

If we use Python coding to crawl Google Trends alone, we will easily encounter the reCAPTHCA verification system. This brings great challenges to our crawling process.

However, Scrapeless Google Trends Scraping API integrates CAPTCHA solver and intelligent IP rotation, so there is no need to worry about being monitored and identified by the website. Scrapeless guarantees a 99.9% website crawling success rate, providing you with a completely stable and safe data crawling environment.

4 typical advantages of Scrapeless

  1. Competitive price
    Scrapeless is not only powerful, but also guarantees a more competitive market price. Scrapeless Google trends scraping API service call pricing starts at $2 per 1k successful requests.
  2. Stability
    Extensive experience and robust systems ensure reliable, uninterrupted scraping with advanced CAPTCHA-solving capabilities.
  3. Speed
    A vast proxy pool guarantees efficient, large-scale scraping without IP blocks or delays.
  4. Cost-Effective
    Proprietary technology minimizes costs, allowing us to offer competitive pricing without compromising quality.
  5. SLAS Guarantee
    Service-level agreements ensure consistent performance and reliability for enterprise needs.

FAQs

Yes, it is completely legal to scrape the global, publicly available Google Trends data. However, please don't damage your site by sending too many requests in a short period of time.

Google Trends is not a complete reflection of search activity. Google Trends filters out certain types of searches, such as searches that are performed by very few people. Trends only shows data for popular terms, so terms with low search volume will show as "0"

No, Google Trends does not yet provide a public API. However, you can access Google Trends data from private APIs in third-party developer tools, such as Scrapeless.

Final Thoughts

Google Trends is a valuable data integration tool that provides keyword analysis and popular search topics by analyzing search queries on search engines. In this article, we show in depth how to scrape Google Trends using Python.

However, scraping Google Trends using python coding always encounters the CAPTCHA obstacle. It makes your data extraction particularly difficult. Although the Google Trends API is not available, the Scrapeless Google Trends API will be your ideal tool!

Sign up now to get a powerful Google Trends scraping tool!

At Scrapeless, we only access publicly available data while strictly complying with applicable laws, regulations, and website privacy policies. The content in this blog is for demonstration purposes only and does not involve any illegal or infringing activities. We make no guarantees and disclaim all liability for the use of information from this blog or third-party links. Before engaging in any scraping activities, consult your legal advisor and review the target website's terms of service or obtain the necessary permissions.

Most Popular Articles

Catalogue