🥳Join the Scrapeless Community and Claim Your Free Trial to Access Our Powerful Web Scraping Toolkit!
Back to Blog

How to Track Google Search Results Using Scrapeless on Pipedream?

Alex Johnson
Alex Johnson

Senior Web Scraping Engineer

03-Jul-2025

You already know — Scrapeless has successfully launched on Pipedream! This significantly enhances Scrapeless's convenience in helping build AI Agents and automate workflows. Simply by using node links and entering simple code, you can complete and repeatedly execute automated programs. In this outstanding tutorial, we’ll attempt to use Scrapeless's Scraping API module to track keyword update results for: Revenged Love.

I can't wait to get started on this article!

👉 Learn more about node usage

Crucial Prerequisites

How to use Scrapeless on Pipedream? With just a few basic setups, you can connect Scrapeless to Pipedream and automatically collect and process data scraped from any search engine.

Obtain Scrapeless API Key

If you don't already have a Scrapeless account:

Obtain Scrapeless API Key

Configure Pipedream

Create a Pipedream account and set up your Scrapeless API Key in Pipedream. Now, go to the "Account" tab in Pipedream and add the key there, as shown below:

Configure Scrpeless API key on Pipedream

Then, set up your API key like this:

set up your API key

Workflow Overview

Everything is ready for you! Here are the nodes we'll use:

TriggerScraping APINodejs (transform the output structure) → OpenAI ChatNodejs (track keywords on Discord)

Step 1. Set Up the Trigger

The key to tracking data is repeated execution. We can use a trigger at the start of the workflow to set the execution interval. Click "Add Trigger" and configure a "Custom Interval." Now, you can set the schedule as needed.

Set Up the Trigger

Step 2. Configure the Scraping API Module

We’ll use the Google Search feature in the Scraping API module to call Deep SerpApi for keyword searches: revenged love. Here are the parameters I used:

  • Query: revenged love
  • Language: en
  • Country: us
Configure the Scraping API Module

Click "Test" to successfully return real-time search results. In order to wait for the results of the asynchronous task, you may need to manually click Resume after submitting the query.

submit the query

Step 3. Add Node.js 20.x Node

Since Google Search returns results in HTML, which isn’t practical for most scenarios, we can add a data transformation node to clean and filter out the content we need, such as:

  • video_results
  • inline_images
Add Node.js 20.x Node

Here's the filtering code to use:

Copy
// import axios from "axios"

export default defineComponent({
  async run({ steps, $ }) {
    // Reference previous step data using the steps object and return data to use it in future steps
    const videos = steps.get_data.$return_value?.video_results || {}
    const result = steps.get_data.$return_value?.inline_images || {}
    return JSON.stringify({
      ...videos,
      ...result
    })
  },
})

Here's the returned result we obtained:

Copy
{"0":{"position":1,"thumbnail":"https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSd0eIXSUrXjBwCh3aU30jL3aVig5lxDtQL9w&s","related_content_id":"CnMxDy61BewRJM,8yBwVXcZCBr9dM","related_content_link":"https://www.google.com/search/about-this-image?img=H4sIAAAAAAAA_-MS5fh88MDqr-vaWqQEup4d_flg6dumN4wAqmApcRcAAAA%3D&q=https://www.viki.com/tv/38371c-first-love-again&ctx=iv&hl=en-US","source":"www.viki.com","source_logo":"","title":"First love, Again | Watch with English Subtitles & More | Viki","link":"https://www.viki.com/tv/38371c-first-love-again","original":"https://1.vikiplatform.com/c/38371c/25a36c57fc.jpg?x=b&a=0x0","original_width":3120,"original_height":1744,"in_stock":false,"is_product":false},"1":{"position":2,"title":"ENG SUB Revenged Love|EP01:Falling for the rival","link":"https://www.youtube.com/watch?v=tlCSMHgOKbE","snippet":"ENG SUB Revenged Love|EP01:Falling for the rival. 1.9M views · 2 weeks ago #RevengedLove #BL ...more. RevengedLove. 307K.","duration":"47:42","rich_snippet":{"top":{"detected_extensions":{},"extensions":[]}},"video_link":"https://encrypted-vtbn0.gstatic.com/video?q=tbn:ANd9GcR2yueHQqTvdFyBeda4_-uoEOQInAJfRm9F9A"},"2":{"position":3,"title":"EP.5 Revenged Love (2025) Engsub - video Dailymotion","link":"https://www.dailymotion.com/video/x9lrrn8","snippet":"Watch EP.5 Revenged Love (2025) Engsub - RJSN on Dailymotion. ... Revenged Love (2025) Ep 6 Eng Sub #RevengedLove · Daily Drama HD™. yesterday.","duration":"44:22","rich_snippet":{"top":{"detected_extensions":{},"extensions":[]}},"video_link":"https://encrypted-vtbn0.gstatic.com/video?q=tbn:ANd9GcS6XBgsSk3vqCefja0bVhQSxavTw43aP5dqJg"},"3":{"position":4,"title":"(2025) REVENGED LOVE EP 8 ENG SUB - video Dailymotion","link":"https://www.dailymotion.com/video/x9m5r3a","snippet":"EP.8 Revenged Love (2025) Engsub · RJSN. yesterday. 1:01:39. Revenged Love (2025) EP 8 ENG SUB · Spirit Studios HD. yesterday. 1:06:11. (2025) ...","duration":"41:17","rich_snippet":{"top":{"detected_extensions":{},"extensions":[]}}}}

Step 4. Add an OpenAI Chat Node

To directly and simply obtain the most important data, you can add an OpenAI Chat node for efficient data consolidation. We need to bind the OpenAI API key and select the target model.

If you expect large amounts of returned data, don’t forget to adjust Max Tokens to avoid failures or timeouts.

Add an OpenAI Chat Node

Discord has strict formatting requirements for data, and for aesthetic tracking, we now need to give Chat clear instructions: return only the target parameters in JSON format without Markdown. This makes it easier for later configuration.

Here’s a reference prompt you can refine:

Copy
Format the following JSON search results into an array of objects for Discord embeds. Each object must include:

- title (string)
- link (string)
- description (string)
- image_url (string)
- source (string)
- duration (string, optional)

❗️Output only raw JSON without any markdown code block, backticks, or explanation. If it is a arrary javascript type, you need output the Array type not a string type

Data:
{{steps.transform_data.$return_value}} // The address where the result is returned

steps.transform_data.$return_value is the array returned from the previous step in your workflow.)

After testing, check the returned result, but ignore unimportant details—focus on the content under generated_message:

output of chat node
Copy
[{"title":"First love, Again | Watch with English Subtitles & More | Viki","link":"https://www.viki.com/tv/38371c-first-love-again","description":"First love, Again | Watch with English Subtitles & More | Viki","image_url":"https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSd0eIXSUrXjBwCh3aU30jL3aVig5lxDtQL9w&s","source":"www.viki.com"},{"title":"ENG SUB Revenged Love|EP01:Falling for the rival","link":"https://www.youtube.com/watch?v=tlCSMHgOKbE","description":"ENG SUB Revenged Love|EP01:Falling for the rival. 1.9M views · 2 weeks ago #RevengedLove #BL ...more. RevengedLove. 307K.","image_url":"https://encrypted-vtbn0.gstatic.com/video?q=tbn:ANd9GcR2yueHQqTvdFyBeda4_-uoEOQInAJfRm9F9A","source":"www.youtube.com","duration":"47:42"},{"title":"EP.5 Revenged Love (2025) Engsub - video Dailymotion","link":"https://www.dailymotion.com/video/x9lrrn8","description":"Watch EP.5 Revenged Love (2025) Engsub - RJSN on Dailymotion. ... Revenged Love (2025) Ep 6 Eng Sub #RevengedLove · Daily Drama HD™. yesterday.","image_url":"https://encrypted-vtbn0.gstatic.com/video?q=tbn:ANd9GcS6XBgsSk3vqCefja0bVhQSxavTw43aP5dqJg","source":"www.dailymotion.com","duration":"44:22"},{"title":"(2025) REVENGED LOVE EP 8 ENG SUB - video Dailymotion","link":"https://www.dailymotion.com/video/x9m5r3a","description":"EP.8 Revenged Love (2025) Engsub · RJSN. yesterday. 1:01:39. Revenged Love (2025) EP 8 ENG SUB · Spirit Studios HD. yesterday. 1:06:11. (2025) ...","image_url":"https://encrypted-vtbn0.gstatic.com/video?q=tbn:ANd9GcS6XBgsSk3vqCefja0bVhQSxavTw43aP5dqJg","source":"www.dailymotion.com","duration":"41:17"}]

Step 5. Track the Keywords Updates on Discord

Congratulations! All the data is ready. Now, we just need to complete the final step: bind Discord and configure the auto-forwarding script. Since Chat returns results in JSON format, we can add a parameter for format conversion: 'string' ? JSON.parse(_res) : _res.

Here's the code:

Copy
import { axios } from "@pipedream/platform";

export default defineComponent({
  async run({ steps, $ }) {
    
    const _res = steps.chat1.$return_value.generated_message.content;
    const results = typeof _res === 'string' ? JSON.parse(_res) : _res

    const discordWebhookUrl = "YOUR_DISCORD_WEBHOOK";

  
    const embeds = results.map(item => ({
      title: item.title,
      url: item.link,
      description: item.description || "No description.",
      color: 0x7289DA, // Discord blurple
      image: item.image_url ? { url: item.image_url } : undefined,
      fields: [
        item.source ? { name: "Source", value: item.source, inline: true } : undefined,
        item.duration ? { name: "Duration", value: item.duration, inline: true } : undefined,
      ].filter(Boolean),
      timestamp: new Date().toISOString(),
    }));

    
    const payload = {
      content: "🔎 **Search Results:**",
      embeds: embeds.slice(0, 10),
    };

    try {
      const response = await axios($, {
        method: "POST",
        url: discordWebhookUrl,
        headers: {
          "Content-Type": "application/json",
        },
        data: payload,
      });

      console.log("Sent to Discord:", response.status);
      return { success: true, count: embeds.length };
    } catch (error) {
      console.error("Error sending to Discord:", error.message);
      return { success: false, error: error.message };
    }
  },
});
  • steps.Code_step.$return_value is the JSON array returned from the previous step.
  • Each data entry generates a Discord embed, including title, link, description, image, source, and timestamp.
  • A maximum of 10 embeds can be sent at once (Discord’s limit). If you have more data, modify it for batch sending.
  • You can change the text in the content field to customize the displayed title or description.

Step 6. Check Discord Results

Click Test, and after execution completes, we can switch to Discord to see the results!

Check Discord Results

The Ending Thoughts

In this tutorial, we successfully demonstrated how to integrate Scrapeless with Pipedream to build a complete workflow for automatically tracking search results for the keyword "Revenged Love." You learned how to:

  • Configure APIs
  • Parse HTML
  • Use OpenAI Chat to organize data
  • Send real-time updates to Discord

Whether you're monitoring brand trends, popular shows, market movements, or building data input channels for your AI Agent, Scrapeless + Pipedream is an indispensable tool combination.

👉 Try Scrapeless on Pipedream now!

At Scrapeless, we only access publicly available data while strictly complying with applicable laws, regulations, and website privacy policies. The content in this blog is for demonstration purposes only and does not involve any illegal or infringing activities. We make no guarantees and disclaim all liability for the use of information from this blog or third-party links. Before engaging in any scraping activities, consult your legal advisor and review the target website's terms of service or obtain the necessary permissions.

Most Popular Articles

Catalogue