🌐

Web Scraping with Crawl 4 AI

Jun 9, 2025

Overview

The lecture explains how to scrape websites using Crawl 4 AI, including single-page and multi-page scraping, integration with FastAPI to build a REST API, and how to process sitemaps for automated large-scale scraping.

Setting Up Crawl 4 AI

  • Install Crawl 4 AI using pip install crawl4ai.
  • Run crawl4ai-s setup to configure headless browsers and necessary dependencies.
  • Headless browsers like Playwright are used for background crawling.

Basic Web Scraping with Crawl 4 AI

  • Use main.py to asynchronously crawl a single website (e.g., www.codingwithroby.com).
  • Crawling fetches page content and outputs it for review.

Multi-page Crawling and Configuration

  • Use main2.py to crawl multiple URLs by setting up a crawl batch.
  • Configure browser settings, cache checks, and robots.txt compliance.
  • Passing a list of URLs allows batch scraping and processing.
  • Outputs include content previews, metadata, and counts of internal/external links per page.

Legal and Ethical Scraping Practices

  • Always check and respect the website's robots.txt for scraping permissions.
  • Ignoring robots.txt can result in IP bans and legal issues.

Building a REST API for Website Crawling

  • Integrate Crawl 4 AI with FastAPI to enable crawling via API endpoints.
  • Use Pydantic for validating and structuring response data.
  • Create an endpoint to accept a URL and return crawl results including metadata and link counts.

Crawling Sitemaps for Automated Scraping

  • Use sitemap.xml to discover and scrape all accessible URLs from a website.
  • Update the FastAPI endpoint to accept a sitemap URL, extract locations, and batch crawl listed pages.
  • Return structured results for each URL in the sitemap.

Key Terms & Definitions

  • Crawl 4 AI — A Python package for automated web scraping using headless browsers.
  • Headless browser — A browser running in the background without a user interface, used for automation.
  • robots.txt — A file that specifies which parts of a website can be crawled by bots.
  • sitemap.xml — An XML file listing all accessible URLs on a website for easier navigation and crawling.
  • FastAPI — A high-performance Python web framework for building APIs.
  • Pydantic — A data validation library used with FastAPI to ensure correct data structure.

Action Items / Next Steps

  • Install Crawl 4 AI and complete the setup process.
  • Experiment with basic and batch crawling scripts provided.
  • Implement API endpoints using FastAPI for single and sitemap-based crawls.
  • Review a website's robots.txt and sitemap.xml before scraping.
  • Optional: Explore integrating crawled data with AI agents or databases.

Full transcript