> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/open-webui/open-webui/llms.txt
> Use this file to discover all available pages before exploring further.

# Voice & Video

> Hands-free voice and video calling with speech-to-text and text-to-speech capabilities

## Overview

Open WebUI provides comprehensive voice and video capabilities, enabling natural spoken interactions with AI models through multiple Speech-to-Text (STT) and Text-to-Speech (TTS) providers.

## Speech-to-Text (STT)

Convert spoken audio to text using various providers.

### Supported Providers

<Tabs>
  <Tab title="Local Whisper">
    **Faster Whisper** - Self-hosted transcription

    * No API costs
    * Privacy-focused (local processing)
    * Multiple model sizes
    * GPU acceleration support
    * VAD filtering

    ```python theme={null}
    {
      "STT_ENGINE": "",
      "WHISPER_MODEL": "base",  // tiny, base, small, medium, large
      "WHISPER_MODEL_AUTO_UPDATE": true
    }
    ```
  </Tab>

  <Tab title="OpenAI Whisper">
    **OpenAI API** - Cloud transcription

    * High accuracy
    * Fast processing
    * Multiple languages
    * Usage-based pricing

    ```python theme={null}
    {
      "STT_ENGINE": "openai",
      "STT_MODEL": "whisper-1",
      "STT_OPENAI_API_BASE_URL": "https://api.openai.com/v1",
      "STT_OPENAI_API_KEY": "sk-..."
    }
    ```
  </Tab>

  <Tab title="Deepgram">
    **Deepgram API** - Real-time transcription

    * Streaming support
    * Speaker diarization
    * Competitive pricing
    * High accuracy

    ```python theme={null}
    {
      "STT_ENGINE": "deepgram",
      "STT_MODEL": "nova-2",
      "DEEPGRAM_API_KEY": "your-api-key"
    }
    ```
  </Tab>

  <Tab title="Azure Speech">
    **Microsoft Azure** - Enterprise transcription

    * Speaker identification
    * Multi-language support
    * Large file support (200MB)
    * Diarization

    ```python theme={null}
    {
      "STT_ENGINE": "azure",
      "AUDIO_STT_AZURE_API_KEY": "your-key",
      "AUDIO_STT_AZURE_REGION": "eastus",
      "AUDIO_STT_AZURE_LOCALES": "en-US,es-ES,fr-FR",
      "AUDIO_STT_AZURE_MAX_SPEAKERS": 3
    }
    ```
  </Tab>

  <Tab title="Mistral">
    **Mistral AI** - Latest voxtral models

    * Voxtral-mini-latest
    * Chat completions API support
    * High accuracy
    * Multi-language

    ```python theme={null}
    {
      "STT_ENGINE": "mistral",
      "STT_MODEL": "voxtral-mini-latest",
      "AUDIO_STT_MISTRAL_API_KEY": "your-key",
      "AUDIO_STT_MISTRAL_API_BASE_URL": "https://api.mistral.ai/v1",
      "AUDIO_STT_MISTRAL_USE_CHAT_COMPLETIONS": false
    }
    ```
  </Tab>
</Tabs>

### Transcription Workflow

<Steps>
  <Step title="Audio Input">
    User records or uploads audio:

    * Microphone capture
    * File upload
    * Supported formats: WAV, MP3, WebM, M4A, FLAC
  </Step>

  <Step title="Preprocessing">
    Audio preparation:

    * Format conversion (if needed)
    * Compression for large files
    * Splitting if exceeds size limits
  </Step>

  <Step title="Transcription">
    Send to configured STT provider:

    * Process in chunks if necessary
    * Apply language settings
    * Handle diarization (Azure)
  </Step>

  <Step title="Result Assembly">
    Combine and format results:

    * Merge chunk transcriptions
    * Clean up text
    * Return to chat interface
  </Step>
</Steps>

### Audio File Processing

<CardGroup cols={2}>
  <Card title="Format Support" icon="file-audio">
    **Accepted formats:**

    * WAV, MP3, WebM
    * M4A, FLAC, MPEG
    * MP4 (audio track)
    * Automatic conversion if needed
  </Card>

  <Card title="Size Limits" icon="weight-scale">
    **Maximum file sizes:**

    * OpenAI/Deepgram/Mistral: 20MB
    * Azure: 200MB
    * Auto-compression if exceeded
    * Intelligent chunking
  </Card>
</CardGroup>

### Compression & Chunking

Automatically handled by Open WebUI:

```python theme={null}
# From routers/audio.py:1109-1168
# Compression
- Frame rate reduced to 16kHz
- Mono conversion
- 32kbps bitrate

# Chunking
- Splits large files intelligently
- Maintains audio quality
- Parallel processing
- Automatic cleanup
```

## Text-to-Speech (TTS)

Convert AI responses to natural-sounding speech.

### Supported Engines

<Tabs>
  <Tab title="OpenAI TTS">
    **OpenAI API** - High-quality voices

    * Multiple voices (alloy, echo, fable, onyx, nova, shimmer)
    * Natural intonation
    * Fast generation
    * API-based

    ```python theme={null}
    {
      "TTS_ENGINE": "openai",
      "TTS_MODEL": "tts-1-hd",
      "TTS_VOICE": "nova",
      "TTS_OPENAI_API_BASE_URL": "https://api.openai.com/v1",
      "TTS_OPENAI_API_KEY": "sk-..."
    }
    ```
  </Tab>

  <Tab title="ElevenLabs">
    **ElevenLabs** - Premium voice cloning

    * Voice cloning
    * Emotional range
    * Multiple languages
    * Custom voices

    ```python theme={null}
    {
      "TTS_ENGINE": "elevenlabs",
      "TTS_MODEL": "eleven_multilingual_v2",
      "TTS_VOICE": "EXAVITQu4vr4xnSDxMaL",
      "TTS_API_KEY": "your-elevenlabs-key"
    }
    ```
  </Tab>

  <Tab title="Azure Speech">
    **Microsoft Azure** - Enterprise TTS

    * 100+ voices
    * 75+ languages
    * SSML support
    * Neural voices

    ```python theme={null}
    {
      "TTS_ENGINE": "azure",
      "TTS_VOICE": "en-US-JennyNeural",
      "TTS_API_KEY": "your-azure-key",
      "TTS_AZURE_SPEECH_REGION": "eastus",
      "TTS_AZURE_SPEECH_OUTPUT_FORMAT": "audio-24khz-48kbitrate-mono-mp3"
    }
    ```
  </Tab>

  <Tab title="Transformers">
    **Local Generation** - Self-hosted

    * Microsoft SpeechT5
    * No API costs
    * Privacy-focused
    * CPU/GPU support

    ```python theme={null}
    {
      "TTS_ENGINE": "transformers",
      "TTS_MODEL": "cmu_us_slt_arctic"  // Speaker embedding
    }
    ```
  </Tab>
</Tabs>

### Speech Generation API

```bash theme={null}
# POST /api/v1/audio/speech
curl -X POST "https://your-instance/api/v1/audio/speech" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Hello, this is a test of text to speech.",
    "voice": "nova",
    "model": "tts-1-hd"
  }'

# Returns MP3 audio file
```

### Voice Selection

Get available voices for configured engine:

```bash theme={null}
# GET /api/v1/audio/voices
{
  "voices": [
    {"id": "alloy", "name": "alloy"},
    {"id": "echo", "name": "echo"},
    {"id": "fable", "name": "fable"},
    {"id": "nova", "name": "nova"},
    {"id": "shimmer", "name": "shimmer"}
  ]
}
```

### Response Caching

Generated speech is cached based on:

* Input text hash
* Engine and model
* Voice selection

Identical requests return cached audio instantly.

## Voice & Video Calling

Real-time voice and video chat with AI models.

### Features

<CardGroup cols={2}>
  <Card title="Hands-Free Mode" icon="microphone">
    Voice-only conversations:

    * Continuous listening
    * Automatic speech detection
    * Voice activity detection (VAD)
    * Wake word support
  </Card>

  <Card title="Video Calling" icon="video">
    Face-to-face AI interaction:

    * Real-time video feed
    * Avatar display
    * Screen sharing
    * Multi-modal input
  </Card>

  <Card title="Multi-Language" icon="language">
    Global communication:

    * Auto language detection
    * Translation support
    * Multi-language voices
    * Accent options
  </Card>

  <Card title="Low Latency" icon="gauge">
    Optimized performance:

    * Streaming transcription
    * Parallel processing
    * Edge caching
    * WebSocket support
  </Card>
</CardGroup>

### Enabling Voice/Video

```python theme={null}
# Configuration
{
  # STT Setup
  "STT_ENGINE": "openai",
  "STT_MODEL": "whisper-1",
  
  # TTS Setup
  "TTS_ENGINE": "openai",
  "TTS_MODEL": "tts-1-hd",
  "TTS_VOICE": "nova",
  
  # Performance
  "TTS_SPLIT_ON": "sentence"  // Split long responses
}
```

## Advanced Configuration

### Text Splitting for TTS

Improve responsiveness by splitting long texts:

```python theme={null}
{
  "TTS_SPLIT_ON": "sentence"  // sentence, paragraph, none
}
```

**Benefits:**

* Faster initial audio playback
* Smoother streaming experience
* Better for long responses

### Language Settings

<Tabs>
  <Tab title="STT Language">
    Configure transcription language:

    ```python theme={null}
    # Whisper local
    WHISPER_LANGUAGE = "en"  // ISO 639-1 code
    WHISPER_MULTILINGUAL = true

    # API-based (sent with request)
    language = "en-US"
    ```
  </Tab>

  <Tab title="TTS Language">
    Voice language selection:

    ```python theme={null}
    # Azure example
    TTS_VOICE = "fr-FR-DeniseNeural"  // French
    TTS_VOICE = "es-ES-ElviraNeural"  // Spanish
    TTS_VOICE = "ja-JP-NanamiNeural"  // Japanese
    ```
  </Tab>
</Tabs>

### Content Type Filtering

Restrict accepted audio formats:

```python theme={null}
{
  "STT_SUPPORTED_CONTENT_TYPES": [
    "audio/wav",
    "audio/mpeg",
    "audio/webm"
  ]
}
```

<Warning>
  Unsupported formats will be rejected at upload, preventing unnecessary processing.
</Warning>

## Permissions

Control access to voice features:

```python theme={null}
{
  "USER_PERMISSIONS": {
    "chat.stt": true,  // Speech-to-text
    "chat.tts": true   // Text-to-speech
  }
}
```

## Performance Optimization

### Whisper Model Selection

Balance quality vs. speed:

<Tabs>
  <Tab title="tiny">
    **Fastest, lowest accuracy**

    * Use for: Quick transcription, low-resource environments
    * Size: \~75MB
    * Speed: Real-time on CPU
  </Tab>

  <Tab title="base">
    **Good balance**

    * Use for: General purpose, moderate accuracy needed
    * Size: \~150MB
    * Speed: Near real-time on CPU
  </Tab>

  <Tab title="small">
    **Better accuracy**

    * Use for: Production environments, higher accuracy
    * Size: \~500MB
    * Speed: 2-3x real-time on CPU
  </Tab>

  <Tab title="medium/large">
    **Best accuracy**

    * Use for: Critical transcription, GPU available
    * Size: 1.5-3GB
    * Speed: Requires GPU for real-time
  </Tab>
</Tabs>

### GPU Acceleration

Enable for Whisper:

```python theme={null}
# Environment
DEVICE_TYPE = "cuda"  // cuda, cpu, mps

# Whisper config
WHISPER_COMPUTE_TYPE = "float16"  // float16, int8, float32
```

### VAD Filtering

Voice Activity Detection for better quality:

```python theme={null}
WHISPER_VAD_FILTER = true
```

**Benefits:**

* Removes silence
* Reduces hallucinations
* Improves accuracy
* Faster processing

## API Reference

### Configuration Endpoints

```bash theme={null}
# Get audio config
GET /api/v1/audio/config

# Update config
POST /api/v1/audio/config/update
{
  "tts": {...},
  "stt": {...}
}
```

### Transcription

```bash theme={null}
# Transcribe audio file
POST /api/v1/audio/transcriptions

Content-Type: multipart/form-data
- file: [audio file]
- language: "en" (optional)
```

Response:

```json theme={null}
{
  "text": "Transcribed text content",
  "filename": "uploaded-file.mp3"
}
```

### Speech Generation

```bash theme={null}
POST /api/v1/audio/speech
{
  "input": "Text to convert to speech",
  "voice": "nova",
  "model": "tts-1-hd"
}

# Returns audio/mpeg file
```

### Models & Voices

```bash theme={null}
# Get available TTS models
GET /api/v1/audio/models

# Get available voices
GET /api/v1/audio/voices
```

## Best Practices

<CardGroup cols={2}>
  <Card title="Choose Right Provider" icon="magnifying-glass">
    **Consider:**

    * Privacy needs (local vs. cloud)
    * Accuracy requirements
    * Budget constraints
    * Language support
    * Latency tolerance
  </Card>

  <Card title="Optimize Audio Quality" icon="waveform">
    **Tips:**

    * Use high-quality microphone
    * Reduce background noise
    * Clear pronunciation
    * Proper audio levels
    * Supported format
  </Card>

  <Card title="Manage Costs" icon="dollar-sign">
    **Strategies:**

    * Use local Whisper when possible
    * Cache common phrases
    * Monitor API usage
    * Set usage quotas
    * Consider hybrid approach
  </Card>

  <Card title="User Experience" icon="user">
    **Enhance UX:**

    * Enable text splitting for TTS
    * Use appropriate voice
    * Match language settings
    * Provide visual feedback
    * Handle errors gracefully
  </Card>
</CardGroup>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Transcription fails or empty result">
    **Check:**

    * Audio file is not silent/empty
    * Format is supported
    * File size within limits
    * API key is valid
    * Language setting correct
    * VAD not filtering entire audio
  </Accordion>

  <Accordion title="Poor transcription quality">
    **Solutions:**

    * Use larger Whisper model
    * Improve audio quality
    * Reduce background noise
    * Specify correct language
    * Disable VAD if over-filtering
    * Try different provider
  </Accordion>

  <Accordion title="TTS voice sounds unnatural">
    **Try:**

    * Different voice option
    * Higher quality model (tts-1-hd vs tts-1)
    * Azure neural voices
    * ElevenLabs for premium quality
    * Adjust SSML (Azure)
  </Accordion>

  <Accordion title="Slow processing">
    **Optimize:**

    * Use GPU for Whisper
    * Reduce audio file size
    * Enable compression
    * Use smaller model
    * Increase timeout settings
    * Check network latency
  </Accordion>
</AccordionGroup>