Jun 12, 2025
The software and tools used in the workflow include: - **Google Sheets**: For storing and managing scraped company data, email addresses, statuses, and ratings. - **Browserless**: A headless browser service used to take screenshots of websites during the scraping process. - **CapCut** and **Screen Studio Pro**: Video editing and screen recording tools used to create and edit short clips for marketing and outreach. - **N8N (Naden)**: An automation platform used to build and run the scraping, rating, and email outreach workflows. - **Mailgun**, **Mandrill**, and **SMTP** services: Explored for sending HTML emails with better deliverability and formatting. - **DailyMotion** and **JumpShare**: Video hosting platforms used to host outreach campaign videos as embedding videos directly in emails was problematic. - **Canva**: Used for creating and editing images and thumbnails for email campaigns. - **YouTube Transcribe API**: Used in a self-hosted setup for transcribing YouTube videos. - **EasyPanel** and **Hookaroo**: Mentioned as part of the tech stack for hosting and managing services like text-to-speech and SSH keys. - **Telegram Bot**: For receiving error notifications from workflow executions. - **Google Maps API**: Used for scraping business data based on zip codes and categories. These tools are integrated into a custom automated lead generation and outreach system.
The Google Maps API is a set of web services provided by Google that allows developers to access and integrate various Google Maps features into their applications. Here’s how it generally works, especially in the context of scraping business data as described: ### How Google Maps API Works 1. **API Requests** Your application sends HTTP requests to Google Maps API endpoints with specific parameters such as location coordinates, search queries, categories, or zip codes. 2. **Authentication** Each request includes an API key that authenticates your application and tracks usage for billing and quota management. 3. **Data Retrieval** The API responds with structured data in JSON or XML format. For example, when searching for businesses, the Places API returns details like business name, address, phone number, website, ratings, and place IDs. 4. **Pagination** If the search results exceed a single page, the API provides a token to fetch the next page of results. 5. **Rate Limits and Quotas** Google enforces limits on the number of requests per second and daily quotas to prevent abuse and ensure fair usage. 6. **Data Usage** The retrieved data can be used to display maps, markers, or to build lead lists, as in your workflow where business info is scraped and stored in Google Sheets. ### Key Google Maps APIs Used for Lead Scraping - **Places API**: To search for businesses by category, location, or keyword and get detailed info. - **Geocoding API**: To convert addresses or zip codes into geographic coordinates. - **Maps JavaScript API**: For embedding interactive maps (less relevant for scraping). - **Distance Matrix API**: To calculate distances between locations (optional). ### Important Considerations - **Terms of Service**: Google Maps API has strict terms about data usage and scraping. Automated scraping outside of the API’s intended use may violate these terms. - **API Key Security**: Keep your API key secure to prevent unauthorized use. - **Quota Management**: Optimize requests to stay within free or paid quota limits. - **Data Accuracy**: The API provides up-to-date and accurate business data, but some fields may be missing or inconsistent. ### In Your Workflow Context - The scraper uses the Google Maps API to query businesses by zip code and category. - It collects business details including website URLs and emails (when available). - The data is stored and deduplicated in Google Sheets. - The workflow handles errors like offline websites or missing emails. - Rate limits are managed by adjusting scraping intervals and zip code selections. If you want, I can explain more about specific API endpoints or how to set up authentication and requests!
Sure! Here’s a **step-by-step beginner-friendly guide** to implement a Google Maps lead scraping process similar to what you’re working on, using Google Maps API and automation tools: — ## Step 1: Get Access to Google Maps API 1. **Create a Google Cloud Account** - Go to [Google Cloud Console](https://console.cloud.google.com/). - Sign up or log in. 2. **Create a New Project** - Click on “Select a project” > “New Project.” - Name your project (e.g., “Lead Scraper”). 3. **Enable Google Maps APIs** - In the Cloud Console, go to “APIs & Services” > “Library.” - Enable the following APIs: - **Places API** (for business search) - **Geocoding API** (optional, for address to coordinates) - **Maps JavaScript API** (optional, for map display) 4. **Create API Key** - Go to “APIs & Services” > “Credentials.” - Click “Create Credentials” > “API Key.” - Restrict the key to your project and APIs for security. — ## Step 2: Understand What Data You Want to Scrape - Decide on **business categories** (e.g., catering, bookkeepers). - Decide on **geographic areas** (e.g., zip codes, cities). - Plan to collect: business name, address, phone, website, email (if available), ratings. — ## Step 3: Make Your First API Request (Test) 1. Use a tool like **Postman** or your browser to test API calls. 2. Example Places API request to search for businesses near a location: ``` https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=52.3676,4.9041&radius=5000&type=restaurant&key=YOUR\_API\_KEY ``` - `location`: latitude,longitude (e.g., Amsterdam) - `radius`: search radius in meters - `type`: business category (e.g., restaurant, accountant) - `key`: your API key 3. The response will be JSON with business details. — ## Step 4: Automate Data Collection Since Google Maps API returns limited results per request, you’ll need to: - Loop through **multiple zip codes or locations**. - Handle **pagination** to get all results. - Store results in a database or spreadsheet (e.g., Google Sheets). — ## Step 5: Set Up Google Sheets for Data Storage - Create a Google Sheet with columns like: - Business Name - Address - Phone - Website - Email - Rating - Status (e.g., scraped, emailed) — ## Step 6: Build the Scraper Workflow Using Automation Tools You can use tools like **N8N**, **Zapier**, or write a custom script in Python/Node.js. ### Using N8N (No-code/Low-code Automation): 1. **Create a new workflow** in N8N. 2. **Add HTTP Request nodes** to call Google Maps Places API for each zip code. 3. **Parse the JSON response** to extract business info. 4. **Add Google Sheets nodes** to append or update rows with scraped data. 5. **Add logic to handle duplicates** by checking website URLs or place IDs. 6. **Schedule the workflow** to run periodically (e.g., every few minutes). — ## Step 7: Scrape Emails and Website Screenshots (Optional) - Use additional tools or APIs to visit business websites and scrape emails. - Use headless browsers (e.g., Browserless) to take screenshots for rating or verification. — ## Step 8: Set Up Email Outreach Automation - Use your scraped emails to send cold outreach emails. - Build email templates (HTML or plain text). - Use email services like Mailgun or Mandrill for sending bulk emails. - Track email status in your Google Sheet. — ## Step 9: Test and Debug - Run your workflow on a small set of zip codes. - Check for duplicates, missing data, or errors. - Adjust scraping intervals to avoid hitting API rate limits. — ## Step 10: Scale and Optimize - Add more zip codes or categories. - Optimize deduplication logic. - Improve email templates and outreach strategy. - Monitor API usage and costs. — # Summary Checklist | Step | Task | Tools/Resources | |—––|———————————––|————————————| | 1 | Get Google Maps API key | Google Cloud Console | | 2 | Define target categories & areas | Your business goals | | 3 | Test API requests | Postman, browser | | 4 | Automate scraping loop | N8N, Zapier, or custom script | | 5 | Store data in Google Sheets | Google Sheets | | 6 | Build workflow to scrape & save | N8N + Google Sheets nodes | | 7 | Scrape emails & screenshots (optional) | Browserless, custom scripts | | 8 | Set up email outreach | Mailgun, Mandrill, email templates | | 9 | Test & debug | Logs, manual checks | | 10 | Scale & optimize | Workflow improvements | — If you want, I can help you with a **simple example workflow** or a **basic script** to get started! Would you like that?
Great! Here’s a **simple example workflow** using **N8N** (a popular no-code automation tool) to scrape business data from Google Maps Places API and save it to Google Sheets. This will get you started without coding. --- # Simple N8N Workflow to Scrape Google Maps Business Data ### Prerequisites: - N8N account or self-hosted instance ([n8n.io](https://n8n.io/)) - Google Cloud API key with Places API enabled - Google Sheets with a sheet prepared to receive data (columns: Name, Address, Website, Rating) --- ## Step 1: Create Google Sheet Make a Google Sheet with columns: | Name | Address | Website | Rating | --- ## Step 2: Set Up N8N Workflow ### 1. **Start Node** - Use a **Manual Trigger** node to start the workflow manually. ### 2. **HTTP Request Node (Google Places API)** - Add an **HTTP Request** node to call the Places API. **Settings:** - Method: `GET` - URL: ``` https://maps.googleapis.com/maps/api/place/nearbysearch/json ``` - Query Parameters: - `location`: `52.3676,4.9041` (Amsterdam coordinates, change as needed) - `radius`: `5000` (5 km radius) - `type`: `restaurant` (change to your target category) - `key`: `YOUR_API_KEY` (replace with your Google API key) ### 3. **Set Node (Extract Data)** - Add a **Set** node to extract and format the data you want. **Example fields to set:** - `Name`: `{{$json["results"][0]["name"]}}` - `Address`: `{{$json["results"][0]["vicinity"]}}` - `Website`: Leave empty for now (Places API doesn’t always provide website in this endpoint) - `Rating`: `{{$json["results"][0]["rating"]}}` *Note:* To handle multiple results, you’ll need to use a **SplitInBatches** node or a **Function** node to loop through all results. ### 4. **Google Sheets Node (Append Data)** - Add a **Google Sheets** node to append rows to your sheet. **Settings:** - Operation: `Append` - Sheet Name: Your sheet name - Map the fields from the Set node to the columns in your sheet. --- ## Step 3: Run and Test - Execute the workflow manually. - Check your Google Sheet for new rows with business data. --- ## Optional Improvements - Add **pagination handling** to get more than 20 results (Google Places API limit per request). - Use a **Function** node to loop through all results in the JSON array. - Add error handling and deduplication logic. - Extend to scrape emails by visiting websites (requires more advanced setup). - Schedule the workflow to run automatically. --- # Example Function Node to Extract All Results Add this **Function** node after the HTTP Request node to output each business as a separate item: ```javascript const results = $json["results"]; return results.map(item => { return { json: { name: item.name, address: item.vicinity, rating: item.rating || null, place_id: item.place_id } }; }); ``` Then connect this to the Google Sheets node. --- If you want, I can help you build this workflow step-by-step in N8N or provide a simple Python script example instead. Which do you prefer?