🎨

Creating Set Extensions with AI Workflows

Apr 17, 2025

Lecture on Set Extensions Using AI Workflows

Introduction

  • Two workflows for creating set extensions:
    1. Allows selection of image areas to add elements, similar to Photoshop's generator field but enhanced.
    2. Integrates a 3D model with consideration for light direction, colors, and style.
  • Excitement about the potential of these workflows to create amazing projects.

Acknowledgments

  • Thanks to viewers and Patreon supporters.
  • Offer of exclusive resources and community access through Patreon.

Workflow 1: 2D Set Extension

Steps:

  1. Frame Preparation

    • Save the frame for set extension from your footage.
    • Option to track footage using tools like After Effects, Da Vinci Resolve, or Blender.
  2. Software and Tools Setup

    • Install ComfyUI and necessary models from GitHub.
    • Configure ComfyUI environment and download models such as Wild Card Turbo and ControlNet.
  3. Workflow Execution

    • Import the frame into ComfyUI.
    • Use the mask editor to define areas for set extension.
    • Add a creative prompt (e.g., spaceship ruins).
  4. Image Generation and Adjustment

    • Adjust settings for image quality and realism (e.g., sun direction, shadows).

Explanation of Workflow Mechanics

  • Image Scaling: Downscale for better composition, then upscale after generation.
  • Prompt Management: Define positive and negative prompts.
  • Masking: Uses blurred masks and line art to blend new and original images.
  • Image Generation: K sampler and adjustments for denoising.

Workflow 2: 3D Matte Painting

Steps:

  1. 3D Camera Tracking

    • Use Blender for 3D camera tracking.
    • Use selected tracking markers to set up the scene.
  2. Modeling and Rendering

    • Simple box modeling for structures.
    • Render depth and line art passes for AI integration.
  3. ComfyUI Integration

    • Import render passes and prompts into ComfyUI.
    • Use models for line art and depth to generate images.
  4. Image Projection and Compositing

    • Use Blender to project generated images onto models.
    • Render and integrate into original footage.

Advanced Techniques and Adjustments

  • Texture Variation: Generate different textures by changing prompts.
  • Line Art Modification: Enhance details by editing line art in image editors.
  • Combining Workflows: Use both workflows for comprehensive set extensions, adding 2D and 3D elements.

Conclusion

  • Encouragement to explore and experiment with these workflows.
  • Invitation to support on Patreon for more resources and community engagement.
  • Call to share creations and feedback.