🎥

High-Level System Design with YouTube Clone

Jul 17, 2024

High-Level System Design with YouTube Clone

Introduction

  • Learn high-level system design by coding a YouTube clone
  • Three key services: upload, watch, and transcoder
  • Concepts covered: chunking, transcoding, FFmpeg, adaptive bitrate streaming (HLS)
  • Instructor: Kti

Importance of System Design

  • Cloning projects are not enough; system design skills are essential
  • High-level system design (HLSD) is often expected in job interviews
  • The uniqueness of this project: detailed implementation and transcoding by oneself
  • Part of the Hands-On High-Level System Design (HHLD) course, which includes three projects: WhatsApp, YouTube, and Zerodha

Project Overview

  • Upload service
  • Watch service
  • Transcoder service
  • Integration with Kafka
  • Use of PostgreSQL database

Key Points

Initial steps

  • Start from core features (upload, watch, transcode)
  • Gradual implementation and enhancement
  • Begin with uploading a basic file then extend with chunking and adaptive bitrate streaming

Frontend Development

  • Use Next.js for the client
  • Install and use react-player to play videos
  • Create components such as UploadForm and handle file uploads

Backend Development

  • Use Node.js for backend services
  • Create APIs for upload, watch, and transcode functionalities

Setting Up Services

  • Upload Service

    • API route: /upload
    • Upload files to S3 in chunked form
    • Use Postman to verify uploads

  • Watch Service

    • API to fetch video metadata from PostgreSQL
    • Serve the video data to the client

Authentication and Authorization

  • Implement OAuth for sign-in and sign-out
  • Integration of NextAuth for Google sign-in
  • Handle user sessions and display relevant user information

Kafka Integration

  • Used for pub-sub mechanism between services
  • Configure Kafka producer in the upload service and consumer in the transcoder service

Transcoding Videos

  • Using ffmpeg for transcoding
  • Dividing videos into chunks and encoding them at different bitrates
  • Adaptive Bitrate Streaming: Implementation using HLS

Adaptive Bitrate Streaming (HLS)

  • What is adaptive bitrate streaming?
  • How HLS works and its benefits
  • Testing the transcoding and HLS implementation

How Everything Works Together

  • Frontend uploads chunked files to the backend
  • Backend uploads files to S3 and metadata to PostgreSQL
  • After upload, services notify Kafka
  • Transcoder service picks up the message, transcodes the video, and uploads back to S3

Conclusion

  • Final HLD Diagram
  • Detailed step-by-step design and implementation
  • Encouragement to join the HHLD course for detailed guidance

Additional Features for Expansion

  • Recommendation engine
  • Content filtering
  • More complex user management and interaction features

Resources

  • Course sign-up details
  • Videos for understanding adaptive bitrate streaming and Kafka

Full transcript