Skip to main content

Overview

Runtime MetaHuman Lip Sync Documentation

Runtime MetaHuman Lip Sync is a plugin that enables real-time, offline, and cross-platform lip sync for both MetaHuman and custom characters. It allows you to animate a character's lips in response to audio input from various sources, including:

The plugin internally generates visemes (visual representations of phonemes) based on the audio input. Since it works directly with audio data rather than text, the plugin supports multilingual input including but not limited to English, Spanish, French, German, Japanese, Chinese, Korean, Russian, Italian, Portuguese, Arabic, and Hindi. Literally any language is supported as the lip sync is generated from audio phonemes rather than language-specific text processing.

The Standard Model produces 14 visemes and performs lip sync animation using a predefined pose asset. In contrast, the Realistic Model (exclusive to MetaHuman characters) generates 250 morph target changes without relying on a predefined pose asset, resulting in significantly more realistic facial animations.

Character Compatibility​

Despite its name, Runtime MetaHuman Lip Sync works with a wide range of characters beyond just MetaHumans:

  • Daz Genesis 8/9 characters
  • Reallusion Character Creator 3/4 (CC3/CC4) characters
  • Mixamo characters
  • ReadyPlayerMe avatars

Animation Standards Support​

  • FACS-based blendshape systems
  • Apple ARKit blendshape standard
  • Preston Blair phoneme sets
  • 3ds Max phoneme systems
  • Any character with custom morph targets for facial expressions

For detailed instructions on using the plugin with non-MetaHuman characters, see the Custom Character Setup Guide.

Animation Preview​

Check out these short animations to see the quality of lip sync animation produced by the plugin across different character types and models:

Realistic Lip Sync Example
Realistic model with MetaHuman character
Standard Lip Sync Example
Standard model with MetaHuman character
Custom Character Lip Sync Example
Standard model with custom character
Custom Character Lip Sync Example
Standard model with custom character

Key Features​

  • Real-time lip sync from microphone input
  • Offline audio processing support
  • Cross-platform compatibility: Windows, Android, Meta Quest
  • Support for multiple character systems and animation standards
  • Flexible viseme mapping for custom characters
  • Universal language support - works with any spoken language through audio analysis

Lip Sync Models​

The plugin offers two lip sync models to suit different project needs:

The standard lip sync model provides efficient, cross-platform performance with broad character compatibility:

  • Works with MetaHumans and all custom character types
  • Optimized for real-time performance on all platforms
  • Lower resource requirements
  • Full compatibility with local TTS (Runtime Text To Speech plugin)
  • Suitable for mobile and VR/AR applications
Extension Plugin Required

To use the Standard (Faster) Model, you need to install an additional extension plugin. See the Prerequisites section for installation instructions.

You can choose the appropriate model based on your project requirements for performance, character compatibility, and visual quality.

TTS Compatibility Note

While both models support various audio input methods, the Realistic model has limited compatibility with local TTS due to ONNX runtime conflicts. For text-to-speech functionality with the Realistic model, external TTS services (OpenAI, ElevenLabs) are recommended.

How It Works​

The plugin processes audio input in the following way:

  1. Audio data is received as float PCM format with specified channels and sample rate
  2. The plugin processes the audio to generate visemes (phonemes)
  3. These visemes drive the lip sync animation using the character's pose asset
  4. The animation is applied to the character in real-time

Quick Start​

Here's a basic setup for enabling lip sync on your character:

  1. For MetaHuman characters, follow the MetaHuman Setup Guide
  2. For custom characters, follow the Custom Character Setup Guide
  3. Set up audio input processing (such as in the Event Graph)
  4. Connect the Blend Runtime MetaHuman Lip Sync node in the Anim Graph
  5. Play audio and see your character speak!

Additional Resources​

📦 Downloads & Links​

🎥 Video Tutorials​

Realistic Model (High-Quality) Tutorials:​

Standard Model Tutorials:​

General Setup:​

💬 Support​