Audio Processing Guide

This guide covers how to set up different audio input methods to feed audio data to your lip sync generators. Make sure you've completed the Setup Guide before proceeding.

Audio Input Processing

You need to set up a method to process audio input. There are several ways to do this depending on your audio source.

Microphone (Real-time)

This approach performs lip sync in real-time while speaking into the microphone:

Standard Model

1. Create a Capturable Sound Wave using Runtime Audio Importer
   • For Linux with Pixel Streaming, use Pixel Streaming Capturable Sound Wave instead
2. Before starting to capture audio, bind to the OnPopulateAudioData delegate
3. In the bound function, call ProcessAudioData from your Runtime Viseme Generator
4. Start capturing audio from the microphone

Copyable nodes.

Realistic Model

The Realistic Model uses the same audio processing workflow as the Standard Model, but with the RealisticLipSyncGenerator variable instead of VisemeGenerator.

Copyable nodes.

Mood-Enabled Realistic Model

The Mood-Enabled Model uses the same audio processing workflow, but with the MoodMetaHumanLipSyncGenerator variable and additional mood configuration capabilities.

Copyable nodes.

Microphone (Playback)

This approach captures audio from a microphone, then plays it back with lip sync:

Standard Model

1. Create a Capturable Sound Wave using Runtime Audio Importer
   • For Linux with Pixel Streaming, use Pixel Streaming Capturable Sound Wave instead
2. Start audio capture from the microphone
3. Before playing back the capturable sound wave, bind to its OnGeneratePCMData delegate
4. In the bound function, call ProcessAudioData from your Runtime Viseme Generator

Copyable nodes.

Realistic Model

The Realistic Model uses the same audio processing workflow as the Standard Model, but with the RealisticLipSyncGenerator variable instead of VisemeGenerator.

Copyable nodes.

Mood-Enabled Realistic Model

The Mood-Enabled Model uses the same audio processing workflow, but with the MoodMetaHumanLipSyncGenerator variable and additional mood configuration capabilities.

Copyable nodes.

Text-to-Speech (Local)

Regular

This approach synthesizes speech from text using local TTS and performs lip sync:

Standard Model

1. Use Runtime Text To Speech to generate speech from text
2. Use Runtime Audio Importer to import the synthesized audio
3. Before playing back the imported sound wave, bind to its OnGeneratePCMData delegate
4. In the bound function, call ProcessAudioData from your Runtime Viseme Generator

Copyable nodes.

Realistic Model

The Realistic Model uses the same audio processing workflow as the Standard Model, but with the RealisticLipSyncGenerator variable instead of VisemeGenerator.

Copyable nodes.

Mood-Enabled Realistic Model

The Mood-Enabled Model uses the same audio processing workflow, but with the MoodMetaHumanLipSyncGenerator variable and additional mood configuration capabilities.

Copyable nodes.

Streaming

This approach uses streaming text-to-speech synthesis with real-time lip sync:

Standard Model

1. Use Runtime Text To Speech to generate streaming speech from text
2. Use Runtime Audio Importer to import the synthesized audio
3. Before playing back the streaming sound wave, bind to its OnGeneratePCMData delegate
4. In the bound function, call ProcessAudioData from your Runtime Viseme Generator

Copyable nodes.

Realistic Model

The Realistic Model uses the same audio processing workflow as the Standard Model, but with the RealisticLipSyncGenerator variable instead of VisemeGenerator.

Copyable nodes.

Mood-Enabled Realistic Model

The Mood-Enabled Model uses the same audio processing workflow, but with the MoodMetaHumanLipSyncGenerator variable and additional mood configuration capabilities.

Copyable nodes.

Text-to-Speech (External APIs)

Regular

This approach uses the Runtime AI Chatbot Integrator plugin to generate synthesized speech from AI services (OpenAI or ElevenLabs) and perform lip sync:

Standard Model

1. Use Runtime AI Chatbot Integrator to generate speech from text using external APIs (OpenAI, ElevenLabs, etc.)
2. Use Runtime Audio Importer to import the synthesized audio data
3. Before playing back the imported sound wave, bind to its OnGeneratePCMData delegate
4. In the bound function, call ProcessAudioData from your Runtime Viseme Generator

Copyable nodes.

Realistic Model

The Realistic Model uses the same audio processing workflow as the Standard Model, but with the RealisticLipSyncGenerator variable instead of VisemeGenerator.

Copyable nodes.

Mood-Enabled Realistic Model

The Mood-Enabled Model uses the same audio processing workflow, but with the MoodMetaHumanLipSyncGenerator variable and additional mood configuration capabilities.

Copyable nodes.

Streaming

This approach uses the Runtime AI Chatbot Integrator plugin to generate synthesized streaming speech from AI services (OpenAI or ElevenLabs) and perform lip sync:

Standard Model

1. Use Runtime AI Chatbot Integrator to connect to streaming TTS APIs (like ElevenLabs Streaming API)
2. Use Runtime Audio Importer to import the synthesized audio data
3. Before playing back the streaming sound wave, bind to its OnGeneratePCMData delegate
4. In the bound function, call ProcessAudioData from your Runtime Viseme Generator

Copyable nodes.

Realistic Model

The Realistic Model uses the same audio processing workflow as the Standard Model, but with the RealisticLipSyncGenerator variable instead of VisemeGenerator.

Copyable nodes.

Mood-Enabled Realistic Model

The Mood-Enabled Model uses the same audio processing workflow, but with the MoodMetaHumanLipSyncGenerator variable and additional mood configuration capabilities.

Copyable nodes.

From Audio File/Buffer

This approach uses pre-recorded audio files or audio buffers for lip sync:

Standard Model

1. Use Runtime Audio Importer to import an audio file from disk or memory
2. Before playing back the imported sound wave, bind to its OnGeneratePCMData delegate
3. In the bound function, call ProcessAudioData from your Runtime Viseme Generator
4. Play the imported sound wave and observe the lip sync animation

Copyable nodes.

Realistic Model

The Realistic Model uses the same audio processing workflow as the Standard Model, but with the RealisticLipSyncGenerator variable instead of VisemeGenerator.

Copyable nodes.

Mood-Enabled Realistic Model

The Mood-Enabled Model uses the same audio processing workflow, but with the MoodMetaHumanLipSyncGenerator variable and additional mood configuration capabilities.

Copyable nodes.

Streaming Audio Buffer

For streaming audio data from a buffer, you need:

Standard Model

1. Audio data in float PCM format (an array of floating-point samples) available from your streaming source (or use Runtime Audio Importer to support more formats)
2. The sample rate and number of channels
3. Call ProcessAudioData from your Runtime Viseme Generator with these parameters as audio chunks become available

Copyable nodes.

Realistic Model

The Realistic Model uses the same audio processing workflow as the Standard Model, but with the RealisticLipSyncGenerator variable instead of VisemeGenerator.

Copyable nodes.

Mood-Enabled Realistic Model

The Mood-Enabled Model uses the same audio processing workflow, but with the MoodMetaHumanLipSyncGenerator variable and additional mood configuration capabilities.

Copyable nodes.

Note: When using streaming audio sources, make sure to manage audio playback timing appropriately to avoid distorted playback. See the Streaming Sound Wave documentation for more information.

Processing Performance Tips

• Chunk Size: Increasing the ProcessingChunkSize configuration option (e.g. to 320, 480, or 640 samples) can noticeably improve latency with minimal impact on quality or responsiveness.

• Model Type: When using Realistic models, switching to the Highly Optimized model type (selected by default) can improve performance. Note that the original model may produce slightly better quality, particularly with noisy audio.

• Buffer Management: The mood-enabled model processes audio in 320-sample frames (20ms at 16kHz). Ensure your audio input timing aligns with this for optimal performance.

• Generator Recreation: For reliable operation with Realistic models, recreate the generator each time you want to feed new audio data after a period of inactivity.

Next Steps

Once you have audio processing set up, you may want to:

• Learn about Configuration options to fine-tune your lip sync behavior
• Add laughter animation for enhanced expressiveness
• Combine lip sync with existing facial animations using the layering techniques described in the Configuration guide