How to use the plugin

The Runtime AI Chatbot Integrator provides two main functionalities: Text-to-Text chat and Text-to-Speech (TTS). Both features follow a similar workflow:

1. Register your API provider token
2. Configure feature-specific settings
3. Send requests and process responses

Register Provider Token

Before sending any requests, register your API provider token using the RegisterProviderToken function.

Blueprint

C++

// Register an OpenAI provider token, as an example
UAIChatbotCredentialsManager::RegisterProviderToken(
    EAIChatbotIntegratorOrgs::OpenAI, 
    TEXT("sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
);

// Register other providers as needed
UAIChatbotCredentialsManager::RegisterProviderToken(
    EAIChatbotIntegratorOrgs::Anthropic, 
    TEXT("sk-ant-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
);

UAIChatbotCredentialsManager::RegisterProviderToken(
    EAIChatbotIntegratorOrgs::DeepSeek, 
    TEXT("sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
);

etc

Text-to-Text Chat Functionality

The plugin supports two chat request modes for each provider:

Non-Streaming Chat Requests

Retrieve the complete response in a single call.

OpenAI

Blueprint

C++

// Example of sending a non-streaming chat request to OpenAI
FChatbotIntegrator_OpenAISettings Settings;
Settings.Messages.Add(FChatbotIntegrator_OpenAIMessage{
EChatbotIntegrator_OpenAIRole::SYSTEM, 
TEXT("You are a helpful assistant.")
});
Settings.Messages.Add(FChatbotIntegrator_OpenAIMessage{
EChatbotIntegrator_OpenAIRole::USER, 
TEXT("What is the capital of France?")
});

UAIChatbotIntegratorOpenAI::SendChatRequestNative(
Settings, 
FOnOpenAIChatCompletionResponseNative::CreateWeakLambda(
this, 
[this](const FString& Response, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    UE_LOG(LogTemp, Log, TEXT("Chat completion response: %s, Error: %d: %s"), 
        *Response, ErrorStatus.bIsError, *ErrorStatus.ErrorMessage);
}
)
);

DeepSeek

Blueprint

C++

// Example of sending a non-streaming chat request to DeepSeek
FChatbotIntegrator_DeepSeekSettings Settings;
Settings.Messages.Add(FChatbotIntegrator_DeepSeekMessage{
EChatbotIntegrator_DeepSeekRole::SYSTEM, 
TEXT("You are a helpful assistant.")
});
Settings.Messages.Add(FChatbotIntegrator_DeepSeekMessage{
EChatbotIntegrator_DeepSeekRole::USER, 
TEXT("What is the capital of France?")
});

UAIChatbotIntegratorDeepSeek::SendChatRequestNative(
Settings, 
FOnDeepSeekChatCompletionResponseNative::CreateWeakLambda(
this, 
[this](const FString& Reasoning, const FString& Content, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    UE_LOG(LogTemp, Log, TEXT("Chat completion reasoning: %s, Content: %s, Error: %d: %s"), 
        *Reasoning, *Content, ErrorStatus.bIsError, *ErrorStatus.ErrorMessage);
}
)
);

Claude

Blueprint

C++

// Example of sending a non-streaming chat request to Claude
FChatbotIntegrator_ClaudeSettings Settings;
Settings.Messages.Add(FChatbotIntegrator_ClaudeMessage{
EChatbotIntegrator_ClaudeRole::SYSTEM, 
TEXT("You are a helpful assistant.")
});
Settings.Messages.Add(FChatbotIntegrator_ClaudeMessage{
EChatbotIntegrator_ClaudeRole::USER, 
TEXT("What is the capital of France?")
});

UAIChatbotIntegratorClaude::SendChatRequestNative(
Settings, 
FOnClaudeChatCompletionResponseNative::CreateWeakLambda(
this, 
[this](const FString& Response, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    UE_LOG(LogTemp, Log, TEXT("Chat completion response: %s, Error: %d: %s"), 
        *Response, ErrorStatus.bIsError, *ErrorStatus.ErrorMessage);
}
)
);

Gemini

Blueprint

C++

// Example of sending a non-streaming chat request to Gemini
FChatbotIntegrator_GeminiSettings Settings;
Settings.Messages.Add(FChatbotIntegrator_GeminiMessage{
EChatbotIntegrator_GeminiRole::USER, 
TEXT("What is the capital of France?")
});

UAIChatbotIntegratorGemini::SendChatRequestNative(
Settings, 
FOnGeminiChatCompletionResponseNative::CreateWeakLambda(
this, 
[this](const FString& Response, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    UE_LOG(LogTemp, Log, TEXT("Chat completion response: %s, Error: %d: %s"), 
        *Response, ErrorStatus.bIsError, *ErrorStatus.ErrorMessage);
}
)
);

Grok

Blueprint

C++

// Example of sending a non-streaming chat request to Grok
FChatbotIntegrator_GrokSettings Settings;
Settings.Messages.Add(FChatbotIntegrator_GrokMessage{
EChatbotIntegrator_GrokRole::SYSTEM, 
TEXT("You are a helpful assistant.")
});
Settings.Messages.Add(FChatbotIntegrator_GrokMessage{
EChatbotIntegrator_GrokRole::USER, 
TEXT("What is the capital of France?")
});

UAIChatbotIntegratorGrok::SendChatRequestNative(
Settings, 
FOnGrokChatCompletionResponseNative::CreateWeakLambda(
this, 
[this](const FString& Reasoning, const FString& Response, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    UE_LOG(LogTemp, Log, TEXT("Chat completion reasoning: %s, Response: %s, Error: %d: %s"), 
        *Reasoning, *Response, ErrorStatus.bIsError, *ErrorStatus.ErrorMessage);
}
)
);

Streaming Chat Requests

Receive response chunks in real-time for a more dynamic interaction.

OpenAI

Blueprint

C++

// Example of sending a streaming chat request to OpenAI
FChatbotIntegrator_OpenAIStreamingSettings Settings;
Settings.Messages.Add(FChatbotIntegrator_OpenAIMessage{
EChatbotIntegrator_OpenAIRole::SYSTEM, 
TEXT("You are a helpful assistant.")
});
Settings.Messages.Add(FChatbotIntegrator_OpenAIMessage{
EChatbotIntegrator_OpenAIRole::USER, 
TEXT("What is the capital of France?")
});

UAIChatbotIntegratorOpenAIStream::SendStreamingChatRequestNative(
Settings, 
FOnOpenAIChatCompletionStreamNative::CreateWeakLambda(
this, 
[this](const FString& ChunkContent, bool IsFinalChunk, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    UE_LOG(LogTemp, Log, TEXT("Streaming chat chunk: %s, IsFinalChunk: %d, Error: %d: %s"), 
        *ChunkContent, IsFinalChunk, ErrorStatus.bIsError, *ErrorStatus.ErrorMessage);
}
)
);

DeepSeek

Blueprint

C++

// Example of sending a streaming chat request to DeepSeek
FChatbotIntegrator_DeepSeekSettings Settings;
Settings.Messages.Add(FChatbotIntegrator_DeepSeekMessage{
EChatbotIntegrator_DeepSeekRole::SYSTEM, 
TEXT("You are a helpful assistant.")
});
Settings.Messages.Add(FChatbotIntegrator_DeepSeekMessage{
EChatbotIntegrator_DeepSeekRole::USER, 
TEXT("What is the capital of France?")
});

UAIChatbotIntegratorDeepSeekStream::SendStreamingChatRequestNative(
Settings, 
FOnDeepSeekChatCompletionStreamNative::CreateWeakLambda(
this, 
[this](const FString& ReasoningChunk, const FString& ContentChunk, 
       bool IsReasoningFinalChunk, bool IsContentFinalChunk, 
       const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    UE_LOG(LogTemp, Log, TEXT("Streaming reasoning: %s, content: %s, Error: %d: %s"), 
        *ReasoningChunk, *ContentChunk, ErrorStatus.bIsError, *ErrorStatus.ErrorMessage);
}
)
);

Claude

Blueprint

C++

// Example of sending a streaming chat request to Claude
FChatbotIntegrator_ClaudeSettings Settings;
Settings.Messages.Add(FChatbotIntegrator_ClaudeMessage{
EChatbotIntegrator_ClaudeRole::SYSTEM, 
TEXT("You are a helpful assistant.")
});
Settings.Messages.Add(FChatbotIntegrator_ClaudeMessage{
EChatbotIntegrator_ClaudeRole::USER, 
TEXT("What is the capital of France?")
});

UAIChatbotIntegratorClaudeStream::SendStreamingChatRequestNative(
Settings, 
FOnClaudeChatCompletionStreamNative::CreateWeakLambda(
this, 
[this](const FString& ChunkContent, bool IsFinalChunk, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    UE_LOG(LogTemp, Log, TEXT("Streaming chat chunk: %s, IsFinalChunk: %d, Error: %d: %s"), 
        *ChunkContent, IsFinalChunk, ErrorStatus.bIsError, *ErrorStatus.ErrorMessage);
}
)
);

Gemini

Blueprint

C++

// Example of sending a streaming chat request to Gemini
FChatbotIntegrator_GeminiSettings Settings;
Settings.Messages.Add(FChatbotIntegrator_GeminiMessage{
EChatbotIntegrator_GeminiRole::USER, 
TEXT("What is the capital of France?")
});

UAIChatbotIntegratorGeminiStream::SendStreamingChatRequestNative(
Settings, 
FOnGeminiChatCompletionStreamNative::CreateWeakLambda(
this, 
[this](const FString& ChunkContent, bool IsFinalChunk, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    UE_LOG(LogTemp, Log, TEXT("Streaming chat chunk: %s, IsFinalChunk: %d, Error: %d: %s"), 
        *ChunkContent, IsFinalChunk, ErrorStatus.bIsError, *ErrorStatus.ErrorMessage);
}
)
);

Grok

Blueprint

C++

// Example of sending a streaming chat request to Grok
FChatbotIntegrator_GrokSettings Settings;
Settings.Messages.Add(FChatbotIntegrator_GrokMessage{
EChatbotIntegrator_GrokRole::SYSTEM, 
TEXT("You are a helpful assistant.")
});
Settings.Messages.Add(FChatbotIntegrator_GrokMessage{
EChatbotIntegrator_GrokRole::USER, 
TEXT("What is the capital of France?")
});

UAIChatbotIntegratorGrokStream::SendStreamingChatRequestNative(
Settings, 
FOnGrokChatCompletionStreamNative::CreateWeakLambda(
this, 
[this](const FString& ReasoningChunk, const FString& ContentChunk, 
       bool IsReasoningFinalChunk, bool IsContentFinalChunk, 
       const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    UE_LOG(LogTemp, Log, TEXT("Streaming reasoning: %s, content: %s, Error: %d: %s"), 
        *ReasoningChunk, *ContentChunk, ErrorStatus.bIsError, *ErrorStatus.ErrorMessage);
}
)
);

Text-to-Speech (TTS) Functionality

Convert text to high-quality speech audio using leading TTS providers. The plugin returns raw audio data (TArray<uint8>) that you can process according to your project's needs.

While the examples below demonstrate audio processing for playback using the Runtime Audio Importer plugin (see audio importing documentation), the Runtime AI Chatbot Integrator is designed to be flexible. The plugin simply returns the raw audio data, giving you complete freedom in how you process it for your specific use case, which might include audio playback, saving to file, further audio processing, transmitting to other systems, custom visualizations, and more.

Non-Streaming TTS Requests

Non-streaming TTS requests return the complete audio data in a single response after the entire text has been processed. This approach is suitable for shorter texts where waiting for the complete audio isn't problematic.

OpenAI TTS

Blueprint

C++

// Example of sending a TTS request to OpenAI
FChatbotIntegrator_OpenAITTSSettings TTSSettings;
TTSSettings.Input = TEXT("Hello, this is a test of text-to-speech functionality.");
TTSSettings.Voice = EChatbotIntegrator_OpenAITTSVoice::NOVA;
TTSSettings.Speed = 1.0f;
TTSSettings.ResponseFormat = EChatbotIntegrator_OpenAITTSFormat::MP3;

UAIChatbotIntegratorOpenAITTS::SendTTSRequestNative(
TTSSettings, 
FOnOpenAITTSResponseNative::CreateWeakLambda(
this, 
[this](const TArray<uint8>& AudioData, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
	if (!ErrorStatus.bIsError)
	{
		// Process the audio data using Runtime Audio Importer plugin
		UE_LOG(LogTemp, Log, TEXT("Received TTS audio data: %d bytes"), AudioData.Num());

		URuntimeAudioImporterLibrary* RuntimeAudioImporter = URuntimeAudioImporterLibrary::CreateRuntimeAudioImporter();
		RuntimeAudioImporter->AddToRoot();
		RuntimeAudioImporter->OnResultNative.AddWeakLambda(this, [this](URuntimeAudioImporterLibrary* Importer, UImportedSoundWave* ImportedSoundWave, ERuntimeImportStatus Status)
		{
			if (Status == ERuntimeImportStatus::SuccessfulImport)
			{
				UE_LOG(LogTemp, Warning, TEXT("Successfully imported audio"));
				// Handle ImportedSoundWave playback
			}
			Importer->RemoveFromRoot();
		});
		RuntimeAudioImporter->ImportAudioFromBuffer(AudioData, ERuntimeAudioFormat::Mp3);
	}
}
)
);

ElevenLabs TTS

Blueprint

C++

// Example of sending a TTS request to ElevenLabs
FChatbotIntegrator_ElevenLabsTTSSettings TTSSettings;
TTSSettings.Text = TEXT("Hello, this is a test of text-to-speech functionality.");
TTSSettings.VoiceID = TEXT("your-voice-id");
TTSSettings.Model = EChatbotIntegrator_ElevenLabsTTSModel::ELEVEN_TURBO_V2;
TTSSettings.OutputFormat = EChatbotIntegrator_ElevenLabsTTSFormat::MP3_44100_128;

UAIChatbotIntegratorElevenLabsTTS::SendTTSRequestNative(
TTSSettings, 
FOnElevenLabsTTSResponseNative::CreateWeakLambda(
this, 
[this](const TArray<uint8>& AudioData, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
	if (!ErrorStatus.bIsError)
	{
		UE_LOG(LogTemp, Log, TEXT("Received TTS audio data: %d bytes"), AudioData.Num());
		// Process audio data as needed
	}
}
)
);

Google Cloud TTS

Blueprint

C++

// Example of getting voices and then sending a TTS request to Google Cloud
// First, get available voices
UAIChatbotIntegratorGoogleCloudVoices::GetVoicesNative(
TEXT("en-US"), // Optional language filter
FOnGoogleCloudVoicesResponseNative::CreateWeakLambda(
this, 
[this](const TArray<FChatbotIntegrator_GoogleCloudVoiceInfo>& Voices, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    if (!ErrorStatus.bIsError && Voices.Num() > 0)
    {
        // Use the first available voice
        const FChatbotIntegrator_GoogleCloudVoiceInfo& FirstVoice = Voices[0];
        UE_LOG(LogTemp, Log, TEXT("Using voice: %s"), *FirstVoice.Name);

        // Now send TTS request with the selected voice
        FChatbotIntegrator_GoogleCloudTTSSettings TTSSettings;
        TTSSettings.Text = TEXT("Hello, this is a test of text-to-speech functionality.");
        TTSSettings.LanguageCode = FirstVoice.LanguageCodes.Num() > 0 ? FirstVoice.LanguageCodes[0] : TEXT("en-US");
        TTSSettings.VoiceName = FirstVoice.Name;
        TTSSettings.AudioEncoding = EChatbotIntegrator_GoogleCloudAudioEncoding::MP3;

        UAIChatbotIntegratorGoogleCloudTTS::SendTTSRequestNative(
            TTSSettings, 
            FOnGoogleCloudTTSResponseNative::CreateWeakLambda(
                this, 
                [this](const TArray<uint8>& AudioData, const FChatbotIntegratorErrorStatus& TTSErrorStatus)
                {
                    if (!TTSErrorStatus.bIsError)
                    {
                        UE_LOG(LogTemp, Log, TEXT("Received TTS audio data: %d bytes"), AudioData.Num());
                        
                        // Process the audio data using Runtime Audio Importer plugin
                        URuntimeAudioImporterLibrary* RuntimeAudioImporter = URuntimeAudioImporterLibrary::CreateRuntimeAudioImporter();
                        RuntimeAudioImporter->AddToRoot();
                        RuntimeAudioImporter->OnResultNative.AddWeakLambda(this, [this](URuntimeAudioImporterLibrary* Importer, UImportedSoundWave* ImportedSoundWave, ERuntimeImportStatus Status)
                        {
                            if (Status == ERuntimeImportStatus::SuccessfulImport)
                            {
                                UE_LOG(LogTemp, Warning, TEXT("Successfully imported audio"));
                                // Handle ImportedSoundWave playback
                            }
                            Importer->RemoveFromRoot();
                        });
                        RuntimeAudioImporter->ImportAudioFromBuffer(AudioData, ERuntimeAudioFormat::Mp3);
                    }
                    else
                    {
                        UE_LOG(LogTemp, Error, TEXT("TTS request failed: %s"), *TTSErrorStatus.ErrorMessage);
                    }
                }
            )
        );
    }
    else
    {
        UE_LOG(LogTemp, Error, TEXT("Failed to get voices: %s"), *ErrorStatus.ErrorMessage);
    }
}
)
);

Azure TTS

Blueprint

C++

// Example of getting voices and then sending a TTS request to Azure
// First, get available voices
UAIChatbotIntegratorAzureGetVoices::GetVoicesNative(
EChatbotIntegrator_AzureRegion::EAST_US,
FOnAzureVoiceListResponseNative::CreateWeakLambda(
this, 
[this](const TArray<FChatbotIntegrator_AzureVoiceInfo>& Voices, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    if (!ErrorStatus.bIsError && Voices.Num() > 0)
    {
        // Use the first available voice
        const FChatbotIntegrator_AzureVoiceInfo& FirstVoice = Voices[0];
        UE_LOG(LogTemp, Log, TEXT("Using voice: %s (%s)"), *FirstVoice.DisplayName, *FirstVoice.ShortName);

        // Now send TTS request with the selected voice
        FChatbotIntegrator_AzureTTSSettings TTSSettings;
        TTSSettings.Text = TEXT("Hello, this is a test of text-to-speech functionality.");
        TTSSettings.VoiceShortName = FirstVoice.ShortName;
        TTSSettings.LanguageCode = FirstVoice.Locale;
        TTSSettings.Region = EChatbotIntegrator_AzureRegion::EAST_US;
        TTSSettings.OutputFormat = EChatbotIntegrator_AzureTTSFormat::AUDIO_16KHZ_32KBITRATE_MONO_MP3;

        UAIChatbotIntegratorAzureTTS::SendTTSRequestNative(
            TTSSettings, 
            FOnAzureTTSResponseNative::CreateWeakLambda(
                this, 
                [this](const TArray<uint8>& AudioData, const FChatbotIntegratorErrorStatus& TTSErrorStatus)
                {
                    if (!TTSErrorStatus.bIsError)
                    {
                        UE_LOG(LogTemp, Log, TEXT("Received TTS audio data: %d bytes"), AudioData.Num());
                        
                        // Process the audio data using Runtime Audio Importer plugin
                        URuntimeAudioImporterLibrary* RuntimeAudioImporter = URuntimeAudioImporterLibrary::CreateRuntimeAudioImporter();
                        RuntimeAudioImporter->AddToRoot();
                        RuntimeAudioImporter->OnResultNative.AddWeakLambda(this, [this](URuntimeAudioImporterLibrary* Importer, UImportedSoundWave* ImportedSoundWave, ERuntimeImportStatus Status)
                        {
                            if (Status == ERuntimeImportStatus::SuccessfulImport)
                            {
                                UE_LOG(LogTemp, Warning, TEXT("Successfully imported audio"));
                                // Handle ImportedSoundWave playback
                            }
                            Importer->RemoveFromRoot();
                        });
                        RuntimeAudioImporter->ImportAudioFromBuffer(AudioData, ERuntimeAudioFormat::Mp3);
                    }
                    else
                    {
                        UE_LOG(LogTemp, Error, TEXT("TTS request failed: %s"), *TTSErrorStatus.ErrorMessage);
                    }
                }
            )
        );
    }
    else
    {
        UE_LOG(LogTemp, Error, TEXT("Failed to get voices: %s"), *ErrorStatus.ErrorMessage);
    }
}
)
);

Streaming TTS Requests

Streaming TTS delivers audio chunks as they're generated, allowing you to process data incrementally rather than waiting for the entire audio to be synthesized. This significantly reduces the perceived latency for longer texts and enables real-time applications. ElevenLabs Streaming TTS also supports advanced chunked streaming functions for dynamic text generation scenarios.

OpenAI Streaming TTS

Blueprint

C++

UPROPERTY()
UStreamingSoundWave* StreamingSoundWave;

UPROPERTY()
bool bIsPlaying = false;

UFUNCTION(BlueprintCallable)
void StartStreamingTTS()
{
    // Create a sound wave for streaming if not already created
    if (!StreamingSoundWave)
    {
        StreamingSoundWave = UStreamingSoundWave::CreateStreamingSoundWave();
        StreamingSoundWave->OnPopulateAudioStateNative.AddWeakLambda(this, [this]()
        {
            if (!bIsPlaying)
            {
                bIsPlaying = true;
                UGameplayStatics::PlaySound2D(GetWorld(), StreamingSoundWave);
            }
        });
    }

    FChatbotIntegrator_OpenAIStreamingTTSSettings TTSSettings;
    TTSSettings.Text = TEXT("Streaming synthesis output begins with a steady flow of data. This data is processed in real-time to ensure consistency.");
    TTSSettings.Voice = EChatbotIntegrator_OpenAIStreamingTTSVoice::ALLOY;
    
    UAIChatbotIntegratorOpenAIStreamTTS::SendStreamingTTSRequestNative(TTSSettings, FOnOpenAIStreamingTTSNative::CreateWeakLambda(this, [this](const TArray<uint8>& AudioData, bool IsFinalChunk, const FChatbotIntegratorErrorStatus& ErrorStatus)
    {
        if (!ErrorStatus.bIsError)
        {
            UE_LOG(LogTemp, Log, TEXT("Received TTS audio chunk: %d bytes"), AudioData.Num());
            StreamingSoundWave->AppendAudioDataFromRAW(AudioData, ERuntimeRAWAudioFormat::Int16, 24000, 1);
        }
    }));
}

ElevenLabs Streaming TTS

ElevenLabs Streaming TTS supports both standard streaming and advanced chunked streaming modes, which provides flexibility for different use cases.

Standard Streaming Mode

Standard streaming mode processes predefined text and delivers audio chunks as they're generated.

Blueprint

C++

UPROPERTY()
UStreamingSoundWave* StreamingSoundWave;

UPROPERTY()
bool bIsPlaying = false;

UFUNCTION(BlueprintCallable)
void StartStreamingTTS()
{
    // Create a sound wave for streaming if not already created
    if (!StreamingSoundWave)
    {
        StreamingSoundWave = UStreamingSoundWave::CreateStreamingSoundWave();
        StreamingSoundWave->OnPopulateAudioStateNative.AddWeakLambda(this, [this]()
        {
            if (!bIsPlaying)
            {
                bIsPlaying = true;
                UGameplayStatics::PlaySound2D(GetWorld(), StreamingSoundWave);
            }
        });
    }

    FChatbotIntegrator_ElevenLabsStreamingTTSSettings TTSSettings;
    TTSSettings.Text = TEXT("Streaming synthesis output begins with a steady flow of data. This data is processed in real-time to ensure consistency.");
    TTSSettings.Model = EChatbotIntegrator_ElevenLabsTTSModel::ELEVEN_TURBO_V2_5;
    TTSSettings.OutputFormat = EChatbotIntegrator_ElevenLabsTTSFormat::MP3_22050_32;
    TTSSettings.VoiceID = TEXT("YOUR_VOICE_ID");
    TTSSettings.bEnableChunkedStreaming = false; // Standard streaming mode
    
    UAIChatbotIntegratorElevenLabsStreamTTS::SendStreamingTTSRequestNative(GetWorld(), TTSSettings, FOnElevenLabsStreamingTTSNative::CreateWeakLambda(this, [this](const TArray<uint8>& AudioData, bool IsFinalChunk, const FChatbotIntegratorErrorStatus& ErrorStatus)
    {
        if (!ErrorStatus.bIsError)
        {
            UE_LOG(LogTemp, Log, TEXT("Received TTS audio chunk: %d bytes"), AudioData.Num());
            StreamingSoundWave->AppendAudioDataFromEncoded(AudioData, ERuntimeAudioFormat::Mp3);
        }
    }));
}

Chunked Streaming Mode

Chunked streaming mode allows you to dynamically append text during synthesis, perfect for real-time applications where text is generated incrementally (e.g., AI chat responses being synthesized as they're generated). To enable this mode, set bEnableChunkedStreaming to true in your TTS settings.

Blueprint

Initial Setup: Set up chunked streaming by enabling the chunked streaming mode in your TTS settings and creating the initial request:

Append Text for Synthesis: Use this node to dynamically add text during an active chunked streaming session. The bContinuousMode parameter controls how the text is processed:

• When bContinuousMode is true: Text is buffered internally until complete sentence boundaries are detected (periods, exclamation marks, question marks). The system automatically extracts complete sentences for synthesis while keeping incomplete text in the buffer. Use this when text arrives in fragments or partial words where sentence completion is uncertain.
• When bContinuousMode is false: Text is processed immediately without buffering or sentence boundary analysis. Each call results in immediate chunk processing and synthesis. Use this when you have pre-formed complete sentences or phrases that don't require boundary detection.

Flush Continuous Buffer: Forces processing of any buffered continuous text, even if no sentence boundary was detected. Useful when you know no more text will be coming for a while:

Set Continuous Flush Timeout: Configures automatic flushing of the continuous buffer when no new text arrives within the specified timeout:

Set to 0 to disable automatic flushing. Recommended values are 1-3 seconds for real-time applications.

Finish Chunked Streaming: Closes the chunked streaming session and marks the current synthesis as final. Always call this when you're done adding text:

C++

UPROPERTY()
UAIChatbotIntegratorElevenLabsStreamTTS* ChunkedTTSRequest;

UPROPERTY()
UStreamingSoundWave* StreamingSoundWave;

UPROPERTY()
bool bIsPlaying = false;

UFUNCTION(BlueprintCallable)
void StartChunkedStreamingTTS()
{
    // Create a sound wave for streaming if not already created
    if (!StreamingSoundWave)
    {
        StreamingSoundWave = UStreamingSoundWave::CreateStreamingSoundWave();
        StreamingSoundWave->OnPopulateAudioStateNative.AddWeakLambda(this, [this]()
        {
            if (!bIsPlaying)
            {
                bIsPlaying = true;
                UGameplayStatics::PlaySound2D(GetWorld(), StreamingSoundWave);
            }
        });
    }

    FChatbotIntegrator_ElevenLabsStreamingTTSSettings TTSSettings;
    TTSSettings.Text = TEXT(""); // Start with empty text in chunked mode
    TTSSettings.Model = EChatbotIntegrator_ElevenLabsTTSModel::ELEVEN_TURBO_V2_5;
    TTSSettings.OutputFormat = EChatbotIntegrator_ElevenLabsTTSFormat::MP3_22050_32;
    TTSSettings.VoiceID = TEXT("YOUR_VOICE_ID");
    TTSSettings.bEnableChunkedStreaming = true; // Enable chunked streaming mode
    
    ChunkedTTSRequest = UAIChatbotIntegratorElevenLabsStreamTTS::SendStreamingTTSRequestNative(
        GetWorld(), 
        TTSSettings, 
        FOnElevenLabsStreamingTTSNative::CreateWeakLambda(this, [this](const TArray<uint8>& AudioData, bool IsFinalChunk, const FChatbotIntegratorErrorStatus& ErrorStatus)
        {
            if (!ErrorStatus.bIsError && AudioData.Num() > 0)
            {
                UE_LOG(LogTemp, Log, TEXT("Received TTS audio chunk: %d bytes"), AudioData.Num());
                StreamingSoundWave->AppendAudioDataFromEncoded(AudioData, ERuntimeAudioFormat::Mp3);
            }
            
            if (IsFinalChunk)
            {
                UE_LOG(LogTemp, Log, TEXT("Chunked streaming session completed"));
                ChunkedTTSRequest = nullptr;
            }
        })
    );
    
    // Now you can append text dynamically as it becomes available
    // For example, from an AI chat response stream:
    AppendTextToTTS(TEXT("Hello, this is the first part of the message. "));
}

UFUNCTION(BlueprintCallable)
void AppendTextToTTS(const FString& AdditionalText)
{
    if (ChunkedTTSRequest)
    {
        // Use continuous mode (true) when text is being generated word-by-word
        // and you want to wait for complete sentences before processing
        bool bContinuousMode = true;
        
        bool bSuccess = ChunkedTTSRequest->AppendTextForSynthesis(AdditionalText, bContinuousMode);
        if (bSuccess)
        {
            UE_LOG(LogTemp, Log, TEXT("Successfully appended text: %s"), *AdditionalText);
        }
    }
}

// Configure continuous text buffering with custom timeout
UFUNCTION(BlueprintCallable)
void SetupAdvancedChunkedStreaming()
{
    if (ChunkedTTSRequest)
    {
        // Set automatic flush timeout to 1.5 seconds
        // Text will be automatically processed if no new text arrives within this timeframe
        ChunkedTTSRequest->SetContinuousFlushTimeout(1.5f);
    }
}

// Example of handling real-time AI chat response synthesis
UFUNCTION(BlueprintCallable)
void HandleAIChatResponseForTTS(const FString& ChatChunk, bool IsStreamFinalChunk)
{
    if (ChunkedTTSRequest)
    {
        if (!IsStreamFinalChunk)
        {
            // Append each chat chunk in continuous mode
            // The system will automatically extract complete sentences for synthesis
            ChunkedTTSRequest->AppendTextForSynthesis(ChatChunk, true);
        }
        else
        {
            // Add the final chunk
            ChunkedTTSRequest->AppendTextForSynthesis(ChatChunk, true);
            
            // Flush any remaining buffered text and finish the session
            ChunkedTTSRequest->FlushContinuousBuffer();
            ChunkedTTSRequest->FinishChunkedStreaming();
        }
    }
}

// Example of immediate chunk processing (bypassing sentence boundary detection)
UFUNCTION(BlueprintCallable)
void AppendImmediateText(const FString& Text)
{
    if (ChunkedTTSRequest)
    {
        // Use continuous mode = false for immediate processing
        // Useful when you have complete sentences or phrases ready
        ChunkedTTSRequest->AppendTextForSynthesis(Text, false);
    }
}

UFUNCTION(BlueprintCallable)
void FinishChunkedTTS()
{
    if (ChunkedTTSRequest)
    {
        // Flush any remaining buffered text
        ChunkedTTSRequest->FlushContinuousBuffer();
        
        // Mark the session as finished
        ChunkedTTSRequest->FinishChunkedStreaming();
    }
}

Key Features of ElevenLabs Chunked Streaming:

• Continuous Mode: When bContinuousMode is true, text is buffered until complete sentence boundaries are detected, then processed for synthesis
• Immediate Mode: When bContinuousMode is false, text is processed immediately as separate chunks without buffering
• Automatic Flush: Configurable timeout processes buffered text when no new input arrives within the specified timeframe
• Sentence Boundary Detection: Detects sentence endings (., !, ?) and extracts complete sentences from buffered text
• Real-time Integration: Supports incremental text input where content arrives in fragments over time
• Flexible Text Chunking: Multiple strategies available (Sentence Priority, Sentence Strict, Size Based) for optimizing synthesis processing

Getting Available Voices

Some TTS providers offer voice listing APIs to discover available voices programmatically.

Google Cloud Voices

Blueprint

C++

// Example of getting available voices from Google Cloud
UAIChatbotIntegratorGoogleCloudVoices::GetVoicesNative(
TEXT("en-US"), // Optional language filter
FOnGoogleCloudVoicesResponseNative::CreateWeakLambda(
this, 
[this](const TArray<FChatbotIntegrator_GoogleCloudVoiceInfo>& Voices, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    if (!ErrorStatus.bIsError)
    {
        for (const auto& Voice : Voices)
        {
            UE_LOG(LogTemp, Log, TEXT("Voice: %s (%s)"), *Voice.Name, *Voice.SSMLGender);
        }
    }
}
)
);

Azure Voices

Blueprint

C++

// Example of getting available voices from Azure
UAIChatbotIntegratorAzureGetVoices::GetVoicesNative(
EChatbotIntegrator_AzureRegion::EAST_US,
FOnAzureVoiceListResponseNative::CreateWeakLambda(
this, 
[this](const TArray<FChatbotIntegrator_AzureVoiceInfo>& Voices, const FChatbotIntegratorErrorStatus& ErrorStatus)
{
    if (!ErrorStatus.bIsError)
    {
        for (const auto& Voice : Voices)
        {
            UE_LOG(LogTemp, Log, TEXT("Voice: %s (%s)"), *Voice.DisplayName, *Voice.Gender);
        }
    }
}
)
);

Error Handling

When sending any requests, it's crucial to handle potential errors by checking the ErrorStatus in your callback. The ErrorStatus provides information about any issues that might occur during the request.

Blueprint

C++

// Example of error handling in a request
UAIChatbotIntegratorOpenAI::SendChatRequestNative(
    Settings, 
    FOnOpenAIChatCompletionResponseNative::CreateWeakLambda(
        this, 
        [this](const FString& Response, const FChatbotIntegratorErrorStatus& ErrorStatus)
        {
            if (ErrorStatus.bIsError)
            {
                // Handle the error
                UE_LOG(LogTemp, Error, TEXT("Chat request failed: %s"), *ErrorStatus.ErrorMessage);
            }
            else 
            {
                // Process the successful response
                UE_LOG(LogTemp, Log, TEXT("Received response: %s"), *Response);
            }
        }
    )
);

Cancelling Requests

The plugin allows you to cancel both text-to-text and TTS requests while they are in progress. This can be useful when you want to interrupt a long-running request or change the conversation flow dynamically.

Blueprint

C++

// Example of cancelling requests
UAIChatbotIntegratorOpenAI* ChatRequest = UAIChatbotIntegratorOpenAI::SendChatRequestNative(
    ChatSettings, 
    ChatResponseCallback
);

// Cancel the chat request at any time
ChatRequest->Cancel();

// TTS requests can be cancelled similarly
UAIChatbotIntegratorOpenAITTS* TTSRequest = UAIChatbotIntegratorOpenAITTS::SendTTSRequestNative(
    TTSSettings, 
    TTSResponseCallback
);

// Cancel the TTS request
TTSRequest->Cancel();

Best Practices

1. Always handle potential errors by checking the ErrorStatus in your callback
2. Be mindful of API rate limits and costs for each provider
3. Use streaming mode for long-form or interactive conversations
4. Consider cancelling requests that are no longer needed to manage resources efficiently
5. Use streaming TTS for longer texts to reduce perceived latency
6. For audio processing, Runtime Audio Importer plugin offers a convenient solution, but you can implement custom processing based on your project needs
7. When using reasoning models (DeepSeek Reasoner, Grok), handle both reasoning and content outputs appropriately
8. Discover available voices using voice listing APIs before implementing TTS features
9. For ElevenLabs chunked streaming: Use continuous mode when text is generated incrementally (like AI responses) and immediate mode for pre-formed text chunks
10. Configure appropriate flush timeouts for continuous mode to balance responsiveness with natural speech flow
11. Choose optimal chunk sizes and send delays based on your application's real-time requirements

Troubleshooting

• Verify your API credentials are correct for each provider
• Check your internet connection
• Ensure any audio processing libraries you use (such as Runtime Audio Importer) are properly installed when working with TTS features
• Verify you're using the correct audio format when processing TTS response data
• For streaming TTS, make sure you're handling audio chunks correctly
• For reasoning models, ensure you're processing both reasoning and content outputs
• Check provider-specific documentation for model availability and capabilities
• For ElevenLabs chunked streaming: Ensure you call FinishChunkedStreaming when done to properly close the session
• For continuous mode issues: Check that sentence boundaries are properly detected in your text
• For real-time applications: Adjust chunk send delays and flush timeouts based on your latency requirements