Lecture Notes on AI Workflow for Film Production

Introduction

• Lady Gwendolyn in shining armor walks towards a crumbling castle.
• Discussion on an efficient AI workflow for creating cinematic images.

Overview of the Workflow

• AI Tools Utilized:
  • Transform sketches into final cinematic images.
  • Use multiple characters and styles across images.
• Applications:
  • Pre-visualization for movie productions.
  • Concept art creation.
  • Generating panels for graphic novels.

Demonstration of the Workflow

1. Creating a Sketch:

   • Use simple tools like Microsoft Paint to create a black and white sketch.
   • Example shot: Knight overlooking a desert with a castle ruin.

2. Image Variation:

   • Change time of day or setting (e.g., desert to flower field).

3. Handling Multiple Characters:

   • Use different colors for characters (e.g., red for knight, green for gnome).
   • This allows Stable Diffusion to apply distinct prompts and create masks for characters.

4. Reference Images:

   • Add reference images to maintain character consistency.
   • Use character model sheets for more advantages.

5. Style Cohesion:

   • Add a reference image for style to ensure color harmony across images.

Steps to Set Up the Workflow

1. Install ConfUI:
   • Powerful interface for Stable Diffusion.
   • Step-by-step guide available on GitHub.
2. Install Necessary Tools:
   • Download and install Git.
   • Install COM4UI Manager from GitHub.
3. Add Custom Nodes and Models:
   • Download storyboarding simple workflow and install missing nodes.
   • Install Wildcard Turbo Model, control net models, IP adapter models, and clip models.

Creating Images with the Workflow

• Set scene dimensions (e.g., 16:9).
• Create prompts for images, changing the style and character emotions as needed.

Advanced Features of the Workflow

• Style reference for images to enhance overall aesthetics.
• Upscaling option for more detail.
• Emotion modification for character expressions.

Short Film Creation

• Workflow applied to create a full short film.
• Story of Gwendolyn and the gnomes:
  • Gwendolyn confronts gnomes, only to face humorous situations.
  • Emphasis on the power of humor and creativity.

Additional Tools Used

• Video Generation:
  • Used LumaLabs Dream Machine for video production.
• Voiceovers and Sound Effects:
  • Generated using Eleven Labs for speech and sound effects.
• Music:
  • Created using UDU for a medieval atmosphere.
• Editing:
  • Final cut in DaVinci Resolve, a recommended free editing tool.

Conclusion

• Importance of storyboarding in filmmaking.
• Encouragement to share creations using the workflow.
• Appreciation for Patreon supporters.