Lecture Notes: High-Level System Design by Coding a YouTube Clone

Course Goals

• Build a YouTube clone to learn high-level system design.
• Gradually add three key services: Upload, Watch, and Transcoder.

Instructor: Ktia

• Course involves practical high-level design concepts:
  • Chunking
  • Transcoding with FFM Peg
  • Adaptive bit rate streaming using HLS

Key Points from the Lecture

Systems Design and Projects

• The importance of system design in modern projects; simple web dev projects are not impressive anymore.
• This course integrates high-level system design with practical coding.

Components of the YouTube Clone Project

1. Initial Steps: Start Simple
   • Focus on core features first: Upload, Watch, Transcode.
   • Implement client-side video playing using react-player.
   • Use S3 (an AWS service) to store and retrieve videos.
2. Upload Service
   • Created using Node.js with a Next.js client.
   • Initial test: Upload a PNG file to S3, then move to small video files.
   • Use Postman for API testing before integrating into the front end.
   • Implement authentication using OAuth (NextAuth for Next.js server-side logic).
3. Progression and Scaling
   • Implement Kafka: Understand producer/consumer model, pub-sub architecture.
   • Test Kafka integration with upload and transcode services as a simple producer/consumer model.
   • Implement large video file chunking on the frontend and test server-side chunk uploads to S3.
4. Multipart Upload
   • Use AWS SDK for chunked, multipart uploads.
   • Follow the three-part process: Initiate upload, upload parts, and complete upload.
   • Test and debug using Postman.
5. Database Integration
   • Use PostgreSQL for metadata (video title, description, author, S3 URL).
   • Create schema with Prisma ORM for database access.
   • Capture and store metadata during the final upload process.
   • Use and build a Watch service to fetch and display video metadata.
6. Authentication and Session Management
   • Integrate Google sign-in using NextAuth.
   • Implement sign-in/sign-out functionality and user session management.
   • Ensure only authenticated users can perform actions like uploading.
7. Adaptive Bitrate Streaming
   • Understand adaptive streaming protocols (HLS and DASH) and implement for transcoding and viewing.
   • Break videos into chunks and support various resolutions.
   • Process involves transcoding chunks into multiple resolutions, creating manifest files, and adaptive streaming.

Code Demonstrations and Key Integrations

1. Transcoding
   • Using ffmpeg for creating multiple resolution video chunks.
   • Generate manifest files (m3u8 for HLS) to describe different resolution videos and their parts.
2. Testing and Debugging
   • Uploading and testing chunked files via Postman and viewing results on S3.
   • Ensuring manifest files and video chunks align correctly for streaming.
3. Database and Kafka Integration
   • Debugging and verifying video metadata storage in PostgreSQL.
   • Setting up Kafka to handle messages between services (Upload and Transcode services).

Final System Design Workflow

• Client-Server Interaction
  • Front-end Next.js client handles video chunking, user authentication (OAuth), and video requests.
• Service Communication and Upload Process
  • Node.js Upload Service processes chunk uploads using multipart upload to S3.
  • Upload Service updates metadata in PostgreSQL, sends Kafka message for Transcoding.
• Transcoding and Adaptive Streaming
  • Transcoder Service reads from S3 based on Kafka message, performing video transcoding to multiple resolutions.
  • Generates and uploads chunked resolution-specific files and corresponds manifest files to S3.
• Watch Service
  • Fetches video metadata from PostgreSQL and provides video links to client for streaming.
  • Supports streaming with adaptive bitrate using the manifest file.

Conclusion

• Functional code and quality content built within classes, with detailed practical steps for implementing high-level system design.
• Course highlights practical integration of coding with system design principles, offering quality guidance and support.

Additional Notes

• Participants in the HHLD course get extensive guidance and community support.
• Recordings of the course and further projects available (e.g., WhatsApp and Zeroda projects).
• Detailed code walkthrough and session-by-session explanations available through course resources.

Suggested Actions

• Review the course material to understand the step-by-step process of building a video platform.
• Practice coding the project components and test each integration thoroughly.
• Explore additional readings and practical exercises on adaptive bitrate streaming, Kafka, and AWS services.