Voice & Video - Open WebUI

Overview

Open WebUI provides comprehensive voice and video capabilities, enabling natural spoken interactions with AI models through multiple Speech-to-Text (STT) and Text-to-Speech (TTS) providers.

Speech-to-Text (STT)

Convert spoken audio to text using various providers.

Supported Providers

Local Whisper
OpenAI Whisper
Deepgram
Azure Speech
Mistral

Faster Whisper - Self-hosted transcription

No API costs
Privacy-focused (local processing)
Multiple model sizes
GPU acceleration support
VAD filtering

{
  "STT_ENGINE": "",
  "WHISPER_MODEL": "base",  // tiny, base, small, medium, large
  "WHISPER_MODEL_AUTO_UPDATE": true
}

OpenAI API - Cloud transcription

High accuracy
Fast processing
Multiple languages
Usage-based pricing

{
  "STT_ENGINE": "openai",
  "STT_MODEL": "whisper-1",
  "STT_OPENAI_API_BASE_URL": "https://api.openai.com/v1",
  "STT_OPENAI_API_KEY": "sk-..."
}

Deepgram API - Real-time transcription

Streaming support
Speaker diarization
Competitive pricing
High accuracy

{
  "STT_ENGINE": "deepgram",
  "STT_MODEL": "nova-2",
  "DEEPGRAM_API_KEY": "your-api-key"
}

Microsoft Azure - Enterprise transcription

Speaker identification
Multi-language support
Large file support (200MB)
Diarization

{
  "STT_ENGINE": "azure",
  "AUDIO_STT_AZURE_API_KEY": "your-key",
  "AUDIO_STT_AZURE_REGION": "eastus",
  "AUDIO_STT_AZURE_LOCALES": "en-US,es-ES,fr-FR",
  "AUDIO_STT_AZURE_MAX_SPEAKERS": 3
}

Mistral AI - Latest voxtral models

Voxtral-mini-latest
Chat completions API support
High accuracy
Multi-language

{
  "STT_ENGINE": "mistral",
  "STT_MODEL": "voxtral-mini-latest",
  "AUDIO_STT_MISTRAL_API_KEY": "your-key",
  "AUDIO_STT_MISTRAL_API_BASE_URL": "https://api.mistral.ai/v1",
  "AUDIO_STT_MISTRAL_USE_CHAT_COMPLETIONS": false
}

Transcription Workflow

Audio Input

User records or uploads audio:

Microphone capture
File upload
Supported formats: WAV, MP3, WebM, M4A, FLAC

Preprocessing

Audio preparation:

Format conversion (if needed)
Compression for large files
Splitting if exceeds size limits

Transcription

Send to configured STT provider:

Process in chunks if necessary
Apply language settings
Handle diarization (Azure)

Result Assembly

Combine and format results:

Merge chunk transcriptions
Clean up text
Return to chat interface

Audio File Processing

Format Support

Accepted formats:

WAV, MP3, WebM
M4A, FLAC, MPEG
MP4 (audio track)
Automatic conversion if needed

Size Limits

Maximum file sizes:

OpenAI/Deepgram/Mistral: 20MB
Azure: 200MB
Auto-compression if exceeded
Intelligent chunking

Compression & Chunking

Automatically handled by Open WebUI:

# From routers/audio.py:1109-1168
# Compression
- Frame rate reduced to 16kHz
- Mono conversion
- 32kbps bitrate

# Chunking
- Splits large files intelligently
- Maintains audio quality
- Parallel processing
- Automatic cleanup

Text-to-Speech (TTS)

Convert AI responses to natural-sounding speech.

Supported Engines

OpenAI TTS
ElevenLabs
Azure Speech
Transformers

OpenAI API - High-quality voices

Multiple voices (alloy, echo, fable, onyx, nova, shimmer)
Natural intonation
Fast generation
API-based

{
  "TTS_ENGINE": "openai",
  "TTS_MODEL": "tts-1-hd",
  "TTS_VOICE": "nova",
  "TTS_OPENAI_API_BASE_URL": "https://api.openai.com/v1",
  "TTS_OPENAI_API_KEY": "sk-..."
}

ElevenLabs - Premium voice cloning

Voice cloning
Emotional range
Multiple languages
Custom voices

{
  "TTS_ENGINE": "elevenlabs",
  "TTS_MODEL": "eleven_multilingual_v2",
  "TTS_VOICE": "EXAVITQu4vr4xnSDxMaL",
  "TTS_API_KEY": "your-elevenlabs-key"
}

Microsoft Azure - Enterprise TTS

100+ voices
75+ languages
SSML support
Neural voices

{
  "TTS_ENGINE": "azure",
  "TTS_VOICE": "en-US-JennyNeural",
  "TTS_API_KEY": "your-azure-key",
  "TTS_AZURE_SPEECH_REGION": "eastus",
  "TTS_AZURE_SPEECH_OUTPUT_FORMAT": "audio-24khz-48kbitrate-mono-mp3"
}

Local Generation - Self-hosted

Microsoft SpeechT5
No API costs
Privacy-focused
CPU/GPU support

{
  "TTS_ENGINE": "transformers",
  "TTS_MODEL": "cmu_us_slt_arctic"  // Speaker embedding
}

Speech Generation API

# POST /api/v1/audio/speech
curl -X POST "https://your-instance/api/v1/audio/speech" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Hello, this is a test of text to speech.",
    "voice": "nova",
    "model": "tts-1-hd"
  }'

# Returns MP3 audio file

Voice Selection

Get available voices for configured engine:

# GET /api/v1/audio/voices
{
  "voices": [
    {"id": "alloy", "name": "alloy"},
    {"id": "echo", "name": "echo"},
    {"id": "fable", "name": "fable"},
    {"id": "nova", "name": "nova"},
    {"id": "shimmer", "name": "shimmer"}
  ]
}

Response Caching

Generated speech is cached based on:

Input text hash
Engine and model
Voice selection

Identical requests return cached audio instantly.

Voice & Video Calling

Real-time voice and video chat with AI models.

Features

Hands-Free Mode

Voice-only conversations:

Continuous listening
Automatic speech detection
Voice activity detection (VAD)
Wake word support

Video Calling

Face-to-face AI interaction:

Real-time video feed
Avatar display
Screen sharing
Multi-modal input

Multi-Language

Global communication:

Auto language detection
Translation support
Multi-language voices
Accent options

Low Latency

Optimized performance:

Streaming transcription
Parallel processing
Edge caching
WebSocket support

Enabling Voice/Video

# Configuration
{
  # STT Setup
  "STT_ENGINE": "openai",
  "STT_MODEL": "whisper-1",
  
  # TTS Setup
  "TTS_ENGINE": "openai",
  "TTS_MODEL": "tts-1-hd",
  "TTS_VOICE": "nova",
  
  # Performance
  "TTS_SPLIT_ON": "sentence"  // Split long responses
}

Advanced Configuration

Text Splitting for TTS

Improve responsiveness by splitting long texts:

{
  "TTS_SPLIT_ON": "sentence"  // sentence, paragraph, none
}

Benefits:

Faster initial audio playback
Smoother streaming experience
Better for long responses

Language Settings

STT Language
TTS Language

Configure transcription language:

# Whisper local
WHISPER_LANGUAGE = "en"  // ISO 639-1 code
WHISPER_MULTILINGUAL = true

# API-based (sent with request)
language = "en-US"

Voice language selection:

# Azure example
TTS_VOICE = "fr-FR-DeniseNeural"  // French
TTS_VOICE = "es-ES-ElviraNeural"  // Spanish
TTS_VOICE = "ja-JP-NanamiNeural"  // Japanese

Content Type Filtering

Restrict accepted audio formats:

{
  "STT_SUPPORTED_CONTENT_TYPES": [
    "audio/wav",
    "audio/mpeg",
    "audio/webm"
  ]
}

Unsupported formats will be rejected at upload, preventing unnecessary processing.

Permissions

Control access to voice features:

{
  "USER_PERMISSIONS": {
    "chat.stt": true,  // Speech-to-text
    "chat.tts": true   // Text-to-speech
  }
}

Performance Optimization

Whisper Model Selection

Balance quality vs. speed:

tiny
base
small
medium/large

Fastest, lowest accuracy

Use for: Quick transcription, low-resource environments
Size: ~75MB
Speed: Real-time on CPU

GPU Acceleration

Enable for Whisper:

# Environment
DEVICE_TYPE = "cuda"  // cuda, cpu, mps

# Whisper config
WHISPER_COMPUTE_TYPE = "float16"  // float16, int8, float32

VAD Filtering

Voice Activity Detection for better quality:

WHISPER_VAD_FILTER = true

Benefits:

Removes silence
Reduces hallucinations
Improves accuracy
Faster processing

API Reference

Configuration Endpoints

# Get audio config
GET /api/v1/audio/config

# Update config
POST /api/v1/audio/config/update
{
  "tts": {...},
  "stt": {...}
}

Transcription

# Transcribe audio file
POST /api/v1/audio/transcriptions

Content-Type: multipart/form-data
- file: [audio file]
- language: "en" (optional)

Response:

{
  "text": "Transcribed text content",
  "filename": "uploaded-file.mp3"
}

Speech Generation

POST /api/v1/audio/speech
{
  "input": "Text to convert to speech",
  "voice": "nova",
  "model": "tts-1-hd"
}

# Returns audio/mpeg file

Models & Voices

# Get available TTS models
GET /api/v1/audio/models

# Get available voices
GET /api/v1/audio/voices

Best Practices

Choose Right Provider

Consider:

Privacy needs (local vs. cloud)
Accuracy requirements
Budget constraints
Language support
Latency tolerance

Optimize Audio Quality

Tips:

Use high-quality microphone
Reduce background noise
Clear pronunciation
Proper audio levels
Supported format

Manage Costs

Strategies:

Use local Whisper when possible
Cache common phrases
Monitor API usage
Set usage quotas
Consider hybrid approach

User Experience

Enhance UX:

Enable text splitting for TTS
Use appropriate voice
Match language settings
Provide visual feedback
Handle errors gracefully

Troubleshooting

Transcription fails or empty result

Check:

Audio file is not silent/empty
Format is supported
File size within limits
API key is valid
Language setting correct
VAD not filtering entire audio

Poor transcription quality

Solutions:

Use larger Whisper model
Improve audio quality
Reduce background noise
Specify correct language
Disable VAD if over-filtering
Try different provider

TTS voice sounds unnatural

Try:

Different voice option
Higher quality model (tts-1-hd vs tts-1)
Azure neural voices
ElevenLabs for premium quality
Adjust SSML (Azure)

Slow processing

Optimize:

Use GPU for Whisper
Reduce audio file size
Enable compression
Use smaller model
Increase timeout settings
Check network latency

​Overview

​Speech-to-Text (STT)

​Supported Providers

​Transcription Workflow

​Audio File Processing

Format Support

Size Limits

​Compression & Chunking

​Text-to-Speech (TTS)

​Supported Engines

​Speech Generation API

​Voice Selection

​Response Caching

​Voice & Video Calling

​Features

Hands-Free Mode

Video Calling

Multi-Language

Low Latency

​Enabling Voice/Video

​Advanced Configuration

​Text Splitting for TTS

​Language Settings

​Content Type Filtering

​Permissions

​Performance Optimization

​Whisper Model Selection

​GPU Acceleration

​VAD Filtering

​API Reference

​Configuration Endpoints

​Transcription

​Speech Generation

​Models & Voices

​Best Practices

Choose Right Provider

Optimize Audio Quality

Manage Costs

User Experience

​Troubleshooting

Overview

Speech-to-Text (STT)

Supported Providers

Transcription Workflow

Audio File Processing

Compression & Chunking

Text-to-Speech (TTS)

Supported Engines

Speech Generation API

Voice Selection

Response Caching

Voice & Video Calling

Features

Enabling Voice/Video

Advanced Configuration

Text Splitting for TTS

Language Settings

Content Type Filtering

Permissions

Performance Optimization

Whisper Model Selection

GPU Acceleration

VAD Filtering

API Reference

Configuration Endpoints

Transcription

Speech Generation

Models & Voices

Best Practices

Troubleshooting