Notes on Lip Syncing with Rhubarb Lip Sync and Blender

Introduction

• Traditional lip syncing used volume to move the mouth, which was not accurate.
• Rhubarb Lip Sync by Daniel S. Wolf: A command-line program that outputs lip sync timings from voice.
• Used in the 2017 game "Thimbleweed Park" by Ron Gilbert.

Getting Started with Rhubarb Lip Sync

1. Download and Setup:

   • Download and unzip Rhubarb Lip Sync.
   • Download the Blender add-on "Rhubarb Lip Sync for Blender" and install it.
   • Locate and configure the Rhubarb executable for your operating system within Blender.

2. Pose Library Creation:

   • Import a character rig into Blender and select the rig.
   • Go to Pose Mode and use MakeHuman runtime tab to observe mouth shapes.
   • Rhubarb requires a pose library setup for lip sync.

3. Creating Poses:

   • Use UV/Image Editor to reference the shape image for Rhubarb.
   • Select and manipulate bones to match the required mouth shapes.
   • Create new poses for each mouth shape (A, B, C, etc.).
   • Adjust poses to be less extreme and more natural.

Using Rhubarb Lip Sync in Blender

• Assign Poses: Assign each mouth shape to a corresponding slot in Rhubarb.
• Sound and Dialogue Files:
  • Load sound files for lip syncing.
  • Recommended to include dialogue text files, even if not word-for-word accurate.

Animation and Editing

• Sequencing:
  • Adjust the video sequence editor to sync audio with animation starting at a specified frame.
  • Add sound files to Blender's sequence editor.
• Keyframes and Mouth Movements:
  • Rhubarb creates keyframes automatically, making the mouth move.
  • Use Graph Editor for further refinement.
  • Use interpolation and smoothing to improve visual transitions between mouth shapes.

Additional Notes

• Limitations:
  • Automated lip sync is not perfect; it can't fully capture mouth closure sounds (e.g., "mmm").
  • Humans focus more on eyes and gestures rather than lips when listening.
• Overall Animation:
  • Full character animation (blinking, gestures) remains crucial.

Conclusion

• Rhubarb Lip Sync offers a more accurate method for 3D lip syncing.
• Combining different methodologies can enhance animation quality.
• Additional tutorials available for further exploration of lip sync techniques.

Note: The instructions referenced additional resources and tools, which should be reviewed for comprehensive understanding and application.