Web Search - Open WebUI

Overview

Open WebUI’s web search integration allows AI models to access real-time information from the internet, enhancing responses with current data, facts, and sources.

Supported Search Providers

Open WebUI integrates with 15+ search providers, offering flexibility for different use cases and privacy preferences.

Self-Hosted Options

SearXNG

Privacy-focused metasearch engine

Aggregates results from multiple sources
Self-hosted, no tracking
Highly customizable
Free and open source

YaCy

Decentralized search

P2P search network
Full control over infrastructure
No central authority

Commercial APIs

Premium
AI-Optimized
Traditional

High-quality paid services:

Google PSE (Programmable Search Engine)
- Reliable, comprehensive results
- Custom search engines
- Usage-based pricing
Brave Search
- Privacy-focused
- Independent index
- Competitive pricing
Kagi
- Premium search quality
- No ads or tracking
- Advanced features

Configuration

Basic Setup

Enable and configure web search through the admin panel:

# From routers/retrieval.py:534-597
{
  "ENABLE_WEB_SEARCH": true,
  "WEB_SEARCH_ENGINE": "searxng",  // Choose your provider
  "WEB_SEARCH_RESULT_COUNT": 5,
  "WEB_SEARCH_CONCURRENT_REQUESTS": 3
}

Provider-Specific Configuration

SearXNG
Google PSE
Brave Search
Tavily
Perplexity

{
  "WEB_SEARCH_ENGINE": "searxng",
  "SEARXNG_QUERY_URL": "http://searxng:8080/search",
  "SEARXNG_LANGUAGE": "en"
}

Deploy SearXNG using Docker for easy self-hosting:

docker run -d -p 8080:8080 searxng/searxng

{
  "WEB_SEARCH_ENGINE": "google_pse",
  "GOOGLE_PSE_API_KEY": "your-api-key",
  "GOOGLE_PSE_ENGINE_ID": "your-engine-id"
}

Setup:

Create a Programmable Search Engine at Google CSE
Get your API key from Google Cloud Console
Note your Search Engine ID

{
  "WEB_SEARCH_ENGINE": "brave",
  "BRAVE_SEARCH_API_KEY": "your-api-key"
}

{
  "WEB_SEARCH_ENGINE": "tavily",
  "TAVILY_API_KEY": "your-api-key"
}

Get your key at Tavily AI

{
  "WEB_SEARCH_ENGINE": "perplexity",
  "PERPLEXITY_API_KEY": "your-api-key",
  "PERPLEXITY_MODEL": "sonar-medium-online",
  "PERPLEXITY_SEARCH_API_URL": "https://api.perplexity.ai"
}

Advanced Settings

Performance

{
  "WEB_SEARCH_CONCURRENT_REQUESTS": 3,
  "WEB_LOADER_CONCURRENT_REQUESTS": 5,
  "WEB_LOADER_TIMEOUT": "30"
}

Security

{
  "WEB_SEARCH_TRUST_ENV": true,
  "ENABLE_WEB_LOADER_SSL_VERIFICATION": true
}

Filtering

{
  "WEB_SEARCH_DOMAIN_FILTER_LIST": [
    "wikipedia.org",
    "github.com"
  ]
}

Processing

{
  "BYPASS_WEB_SEARCH_WEB_LOADER": false,
  "BYPASS_WEB_SEARCH_EMBEDDING_AND_RETRIEVAL": false
}

Web Content Loading

Loading Engines

Choose how web pages are fetched and processed:

Default
Playwright
Firecrawl
External

Standard HTTP requests

Fast and lightweight
Works for most sites
No JavaScript execution

{
  "WEB_LOADER_ENGINE": "default"
}

Browser automation

JavaScript rendering
Handles dynamic content
Screenshot capabilities

{
  "WEB_LOADER_ENGINE": "playwright",
  "PLAYWRIGHT_WS_URL": "ws://playwright:3000",
  "PLAYWRIGHT_TIMEOUT": 30000
}

Specialized scraping

Advanced extraction
Anti-bot bypass
Rich metadata

{
  "WEB_LOADER_ENGINE": "firecrawl",
  "FIRECRAWL_API_KEY": "your-key",
  "FIRECRAWL_API_BASE_URL": "https://api.firecrawl.dev",
  "FIRECRAWL_TIMEOUT": "60"
}

Custom loader service

Bring your own scraper
Custom processing logic
API-based integration

{
  "WEB_LOADER_ENGINE": "external",
  "EXTERNAL_WEB_LOADER_URL": "http://your-loader:8000",
  "EXTERNAL_WEB_LOADER_API_KEY": "your-key"
}

Using Web Search in Chat

Basic Usage

Trigger web search by mentioning URLs or topics:

# Search and cite sources
What are the latest developments in quantum computing?

# Direct URL loading
Summarize https://example.com/article

# Multiple URLs
Compare these articles:
- https://source1.com
- https://source2.com

Search Workflow

Query Detection

AI determines if web search would enhance the response

Search Execution

Query sent to configured search provider

Returns top N results (configurable)
Concurrent requests for multiple queries

Content Fetching

Web loader retrieves page content

Parallel loading of multiple URLs
Timeout protection
SSL verification

Processing

Content prepared for RAG:

Text extraction
Chunking (if enabled)
Embedding generation (if enabled)

Response Generation

AI uses retrieved content to formulate answer

Cites sources
Combines multiple perspectives
Provides attribution

YouTube Integration

Extract and search YouTube video transcripts:

{
  "YOUTUBE_LOADER_LANGUAGE": ["en", "es", "fr"],
  "YOUTUBE_LOADER_PROXY_URL": "http://proxy:8080",
  "YOUTUBE_LOADER_TRANSLATION": "en"
}

Features:

Automatic transcript extraction
Multi-language support
Translation capabilities
Proxy support for restricted regions

Usage:

Summarize this video: https://youtube.com/watch?v=VIDEO_ID

Domain Filtering

Control which domains are searched:

{
  "WEB_SEARCH_DOMAIN_FILTER_LIST": [
    "wikipedia.org",       # Encyclopedic content
    "github.com",          # Code repositories
    "stackoverflow.com",   # Technical Q&A
    "arxiv.org"            # Academic papers
  ]
}

When filter list is populated, only results from these domains are returned. Leave empty to search all domains.

Performance Optimization

Concurrent Requests

Balance speed and resource usage:

{
  "WEB_SEARCH_CONCURRENT_REQUESTS": 3,    // Parallel searches
  "WEB_LOADER_CONCURRENT_REQUESTS": 5     // Parallel page loads
}

Recommendations:

Small deployments: 2-3 concurrent searches, 3-5 loaders
Medium deployments: 3-5 concurrent searches, 5-10 loaders
Large deployments: 5-10 concurrent searches, 10-20 loaders

Timeouts

{
  "WEB_LOADER_TIMEOUT": "30",           // Seconds
  "PLAYWRIGHT_TIMEOUT": 30000,          // Milliseconds
  "FIRECRAWL_TIMEOUT": "60"             // Seconds
}

Set reasonable timeouts to prevent resource exhaustion. Slow sites can hang requests indefinitely.

Bypass Options

Skip Web Loader

Use search results without fetching full content:

{
  "BYPASS_WEB_SEARCH_WEB_LOADER": true
}

When to use:

Search summaries sufficient
Reduce API calls/bandwidth
Faster responses needed

Skip Embedding

Disable RAG processing for web content:

{
  "BYPASS_WEB_SEARCH_EMBEDDING_AND_RETRIEVAL": true
}

When to use:

Direct content injection
Avoid embedding costs
Real-time freshness required

External Search Integration

Connect custom search services:

{
  "WEB_SEARCH_ENGINE": "external",
  "EXTERNAL_WEB_SEARCH_URL": "http://your-search-api:8000",
  "EXTERNAL_WEB_SEARCH_API_KEY": "your-key"
}

API Contract:

// Request
POST /search
{
  "queries": ["search query 1", "query 2"],
  "count": 5
}

// Response
{
  "results": [
    {
      "title": "Result Title",
      "url": "https://example.com",
      "content": "Snippet..."
    }
  ]
}

API Endpoints

Configuration

# Get web search config
GET /api/v1/retrieval/config

# Response includes web search settings
{
  "web": {
    "ENABLE_WEB_SEARCH": true,
    "WEB_SEARCH_ENGINE": "searxng",
    ...
  }
}

Update Configuration

POST /api/v1/retrieval/config/update
{
  "web": {
    "ENABLE_WEB_SEARCH": true,
    "WEB_SEARCH_ENGINE": "brave",
    "BRAVE_SEARCH_API_KEY": "new-key"
  }
}

Best Practices

Choose Right Provider

Consider:

Privacy requirements
Budget constraints
Result quality needs
Geographic coverage
API rate limits

Optimize Result Count

Balance:

More results = better coverage
Fewer results = faster responses
Recommended: 3-7 results
Adjust based on use case

Use Domain Filtering

Benefits:

Improve result quality
Reduce irrelevant content
Focus on trusted sources
Faster processing

Monitor Costs

Track:

API usage per provider
Bandwidth consumption
Processing time
Set up alerts for limits

Troubleshooting

No search results returned

Check:

Provider API key is valid
Search engine is properly configured
Rate limits not exceeded
Domain filter not too restrictive
Network connectivity to provider

Slow web loading

Solutions:

Reduce WEB_LOADER_TIMEOUT
Increase concurrent requests
Use faster web loader engine
Enable BYPASS_WEB_SEARCH_WEB_LOADER
Check network latency

SSL certificate errors

Options:

Ensure ENABLE_WEB_LOADER_SSL_VERIFICATION is true
Update CA certificates
Configure proxy with valid certs
For internal sites, use custom CA

YouTube videos not working

Verify:

Video has captions/transcript
Language is in YOUTUBE_LOADER_LANGUAGE
Proxy configured if in restricted region
Video is publicly accessible

Security Considerations

Important security practices:

SSL Verification: Always enable in production
API Keys: Use environment variables, never commit
Domain Filtering: Prevent access to internal networks
Rate Limiting: Implement to prevent abuse
Timeout Settings: Prevent resource exhaustion

URL Validation

Open WebUI validates URLs to prevent SSRF attacks:

Blocks private IP ranges (10.0.0.0/8, 192.168.0.0/16, etc.)
Prevents localhost access
Validates URL format
Enforces timeout limits

​Overview

​Supported Search Providers

​Self-Hosted Options

SearXNG

YaCy

​Commercial APIs

​Configuration

​Basic Setup

​Provider-Specific Configuration

​Advanced Settings

Performance

Security

Filtering

Processing

​Web Content Loading

​Loading Engines

​Using Web Search in Chat

​Basic Usage

​Search Workflow

​YouTube Integration

​Domain Filtering

​Performance Optimization

​Concurrent Requests

​Timeouts

​Bypass Options

​Skip Web Loader

​Skip Embedding

​External Search Integration

​API Endpoints

​Configuration

​Update Configuration

​Best Practices

Choose Right Provider

Optimize Result Count

Use Domain Filtering

Monitor Costs

​Troubleshooting

​Security Considerations

​URL Validation

Overview

Supported Search Providers

Self-Hosted Options

Commercial APIs

Configuration

Basic Setup

Provider-Specific Configuration

Advanced Settings

Web Content Loading

Loading Engines

Using Web Search in Chat

Basic Usage

Search Workflow

YouTube Integration

Domain Filtering

Performance Optimization

Concurrent Requests

Timeouts

Bypass Options

Skip Web Loader

Skip Embedding

External Search Integration

API Endpoints

Configuration

Update Configuration

Best Practices

Troubleshooting

Security Considerations

URL Validation