Runway's Gen 3 AI Model for Text to Video

Jul 4, 2024

Runway's Gen 3 AI Model for Text to Video

Overview

  • Company: Runway
  • Model: Gen 3
  • Previous Models: Gen 1 and Gen 2
  • Key Feature: Converts text prompts into video clips
  • New Feature: Lip sync - characters can read scripted text using AI voices

Access and Pricing

  • Website: runwayml.com
  • Availability: Open to everyone
  • Subscription: Starting at $15/month
  • Credit System:
    • 10 credits per second of video generation
    • 5-second video: 50 credits
    • 10-second video: 100 credits

Using Gen 3

  1. Website Navigation

    • Visit runwayml.com and create an account
    • Click 'Try Gen 3' (currently in Alpha)
    • Log in with your account
    • Navigate to the Gen 3 page

  2. Prompting Guide

    • Prompts often consist of three parts:
      1. Camera movement
      2. Scene description
      3. Additional details (e.g., lighting)
    • Example: “Low angle static shot of a woman wearing orange”
    • Manual descriptions required for camera settings, lighting, etc.
    • Use AI tools like ChatGPT or Claude to help generate prompts

  3. Creating Videos

    • Enter the prompt into the specified box
    • Choose video length (5 or 10 seconds)
    • Generate video (time varies between 60-120 seconds)
    • Download or save the video once generated

Settings and Options

  • Watermark: Option to remove Runway watermark
  • Seed Setting: Maintain consistency in generated frames if enabled
  • Custom Presets: Save frequently used prompts for easy access

Results and Limitations

  • Consistency: Mixed results - one out of three might be satisfactory
  • Experimentation: Necessary due to variability in output
  • Lip Sync Quality: Basic, better alternatives available (like HeyGen)

Example Generations

  • Submarine in Deep Ocean: Generated clip looks realistic
  • Detailed Prompt: May result in awkward motion or unrealistic features

Additional Resources

  • Course: “Making Movies with AI” on Skill Leap
    • Combines ChatGPT, MidJourney, Gen 2, and Gen 3 to create films
    • Includes a free trial with 22 video tutorials

Conclusion

  • Gen 3 is a promising yet developing tool in text to video conversion with significant potential improvements ahead.