Oktivo Backend Docs

Appearance

🎯 Add HuggingFace Model Support for STT, TTS & LLM in Agent Pipeline ​

Priority: Medium | Type: Feature | Estimated Effort: 3-5 days ​

Context ​

We currently support a limited set of STT/TTS/LLM providers (Deepgram, Cartesia, ElevenLabs, Murf, Google, OpenAI, Sarvam). We want to expand our model testing capabilities by integrating HuggingFace the largest open-source AI model hub so we can benchmark different models for phone call agent quality, latency, and cost. Especially for Hindi language support.

This issue covers everything you need to know about HuggingFace, how the AI industry uses OpenAI-compatible APIs, and how to implement this cleanly in our existing provider architecture.

Background Knowledge (Read Before Starting) ​

What is HuggingFace? ​

HuggingFace (https://huggingface.co) is the largest platform for open-source AI models. Think of it as "GitHub for AI models". Key things to understand:

ConceptWhat it isAnalogy
HuggingFace HubRepository hosting 800K+ AI models (LLMs, STT, TTS, image, etc.)Like npm/PyPI but for AI models
Transformers libraryPython library to download and run models locallyLike TensorFlow/PyTorch wrapper
Inference API (Serverless)Free/paid API to run models without your own GPU. Rate-limited on free tierLike a cloud function for AI
Inference Endpoints (Dedicated)Deploy a model on a dedicated GPU server you controlLike AWS EC2 but for AI models
Text Generation Inference (TGI)HuggingFace's optimized server for running LLMs. Exposes an OpenAI-compatible /v1/chat/completions endpointLike vLLM or Ollama
HF API TokenYour authentication key (starts with hf_)Like an API key for any service

What is the "OpenAI-Compatible API" Pattern? ​

This is an industry standard you must understand. OpenAI defined the API format for chat completions (/v1/chat/completions), and now almost every LLM provider copies this exact format. This means:

  • You write code for OpenAI's API once
  • You swap base_url and api_key to use a completely different provider
  • The request/response format stays the same

Providers that follow this pattern:

  • HuggingFace TGI, Groq, DeepSeek, Fireworks AI, Together AI, OpenRouter, Ollama, vLLM, Cerebras, Perplexity, xAI, Mistral, and 50+ more

LiveKit's OpenAI plugin already supports this every openai.LLM(), openai.STT(), and openai.TTS() class accepts a base_url parameter to point to any compatible endpoint.

python
# OpenAI (default)
llm = openai.LLM(model="gpt-4o-mini")

# Same code, different provider just change base_url + api_key
llm = openai.LLM(
    model="meta-llama/Llama-3.3-70B-Instruct",
    base_url="https://router.huggingface.co/v1",
    api_key="hf_xxxxx"
)

HuggingFace Products We'll Use ​

1. Serverless Inference API (for LLM) ​

  • URL: https://router.huggingface.co/v1 (OpenAI-compatible)
  • Auth: Authorization: Bearer hf_YOUR_TOKEN
  • Free tier: Rate-limited, good for testing
  • Pro tier ($9/mo): Higher rate limits
  • Models: Llama 3.3, Mistral, Qwen, DeepSeek, Gemma, etc.
  • Docs: https://huggingface.co/docs/api-inference/

2. Serverless Inference API (for STT) ​

  • URL: https://api-inference.huggingface.co/models/{model_id}
  • NOT OpenAI-compatible sends raw audio bytes, returns JSON {"text": "..."}
  • Models: openai/whisper-large-v3-turbo, vasista22/whisper-hindi-large-v2, etc.
  • Limitation: Not streaming sends full audio chunk, gets full text back

3. Serverless Inference API (for TTS) ​

  • URL: https://api-inference.huggingface.co/models/{model_id}
  • NOT OpenAI-compatible sends text, returns raw audio bytes
  • Models: facebook/mms-tts-hin (Hindi), ai4bharat/indic-parler-tts (Hindi), Qwen/Qwen3-TTS, etc.
  • Limitation: Not streaming sends full text, gets full audio back

4. Inference Endpoints (Dedicated for production) ​

Key Python Libraries ​

LibraryPurposeInstall
huggingface_hubOfficial HF Python client. Handles auth, API calls, model downloadspip install huggingface_hub
openaiOpenAI Python client. Works with any OpenAI-compatible endpoint including HFAlready installed (via livekit-agents)
aiohttpAsync HTTP client. Use for raw STT/TTS API calls to HFAlready installed
transformersRun models locally (NOT needed for API-based approach)Don't install unless doing local inference

Important: For our use case, we primarily need huggingface_hub for the STT/TTS raw API calls, and the existing openai library (via LiveKit's OpenAI plugin) for LLM. We do NOT need transformers since we're calling APIs, not running models locally.

What to Implement ​

Task 1: HuggingFace LLM Provider ​

Difficulty: Easy HuggingFace LLM API is OpenAI-compatible, so we use LiveKit's existing OpenAI plugin.

File: Agent/src/providers/llm_providers.py

Add a HuggingFace class to the existing registry:

python
import os
from livekit.plugins import openai, google

class HuggingFaceLLM:
    def __init__(self, model="meta-llama/Llama-3.3-70B-Instruct", temperature=0.2, max_tokens=200):
        self.model = model
        self.temperature = temperature
        self.max_tokens = max_tokens

    def build(self):
        return openai.LLM(
            model=self.model,
            base_url="https://router.huggingface.co/v1",
            api_key=os.getenv("HUGGINGFACE_API_KEY"),
            temperature=self.temperature,
            max_completion_tokens=self.max_tokens
        )

# Add to registry
LLM_REGISTRY = {
    "openai": OpenAI,
    "google": Google,
    "huggingface": HuggingFaceLLM,
}

Hindi-capable LLM models to test:

  • meta-llama/Llama-3.3-70B-Instruct good multilingual, Hindi included
  • Qwen/Qwen2.5-72B-Instruct strong Hindi
  • google/gemma-2-27b-it decent Hindi
  • mistralai/Mistral-Small-24B-Instruct-2501 fast, decent Hindi

Task 2: HuggingFace STT Provider (Custom Plugin) ​

Difficulty: Medium HuggingFace STT API is NOT OpenAI-compatible. We need to write a custom LiveKit STT class.

File: Agent/src/providers/stt_providers.py

HuggingFace STT API works like this:

POST https://api-inference.huggingface.co/models/{model_id}
Headers: Authorization: Bearer hf_xxxxx
Body: raw audio bytes (wav/flac/mp3)
Response: {"text": "transcribed text here"}

You need to implement a class that:

  1. Extends LiveKit's stt.STT base class
  2. Receives audio frames from the LiveKit pipeline
  3. Buffers audio (HF API is not streaming it needs complete audio chunks)
  4. Sends audio to HuggingFace API
  5. Returns SpeechEvent objects that LiveKit understands

Key considerations:

  • HuggingFace STT is NOT streaming it processes complete audio chunks, not real-time streams
  • You MUST use LiveKit's StreamAdapter + silero VAD to buffer speech segments before sending to HF
  • Audio format: HF expects wav/flac bytes. LiveKit provides raw PCM frames you'll need to convert
  • Latency will be higher than Deepgram (which is streaming). This is expected for testing purposes

Follow the existing BaseSTT pattern in stt_providers.py. The custom STT class should be a separate file (e.g., Agent/src/providers/custom/huggingface_stt.py) that gets imported into the registry.

Hindi STT models to test:

  • openai/whisper-large-v3-turbo best overall, supports Hindi
  • vasista22/whisper-hindi-large-v2 fine-tuned specifically for Hindi
  • ai4bharat/indicwhisper-large-v2-hi Indian language specialist
  • openai/whisper-large-v3 highest accuracy, slower

Reference: Look at how LiveKit's own StreamAdapter works for non-streaming STT: https://docs.livekit.io/agents/v1/integrations/stt/ (see "VAD and StreamAdapter" section)

Task 3: HuggingFace TTS Provider (Custom Plugin) ​

Difficulty: Medium-Hard HuggingFace TTS API is NOT OpenAI-compatible. Needs custom LiveKit TTS class.

File: Agent/src/providers/tts_providers.py

HuggingFace TTS API works like this:

POST https://api-inference.huggingface.co/models/{model_id}
Headers: Authorization: Bearer hf_xxxxx
Body: {"inputs": "Text to speak"}
Response: raw audio bytes (wav/flac)

You need to implement a class that:

  1. Extends LiveKit's tts.TTS base class
  2. Receives text from the LLM output
  3. Sends text to HuggingFace API
  4. Converts returned audio bytes to SynthesizedAudio frames that LiveKit can play
  5. Handles chunking for long text, split into sentences and synthesize each chunk

Key considerations:

  • HuggingFace TTS is NOT streaming it generates the full audio at once
  • Latency will be higher than Cartesia (which streams audio). This is expected for testing
  • Audio format: HF returns wav/flac. LiveKit needs raw PCM frames at the right sample rate you'll need to decode
  • For sentence chunking, use LiveKit's built-in tts_text_transforms or implement your own sentence splitter
  • Some HF TTS models have a max input length (usually ~500 chars). Handle this gracefully

Follow the existing BaseTTS pattern in tts_providers.py. Put the custom class in Agent/src/providers/custom/huggingface_tts.py.

Hindi TTS models to test:

  • facebook/mms-tts-hin Meta's Massively Multilingual Speech, good Hindi
  • ai4bharat/indic-parler-tts Indian language specialist, natural sounding
  • Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice newer, high quality
  • parler-tts/parler-tts-large-v1 good quality, English + some Hindi

Task 4: Environment & Config Updates ​

Files to update:

  1. .env Add:

    HUGGINGFACE_API_KEY=hf_xxxxxxxxxxxxxxxxxxxxxxxxx

  2. requirements.txt Add:

    huggingface_hub>=0.25.0

    (The openai library is already installed via livekit-agents[openai])

  3. agent_meta_schema.py No changes needed. The existing stt_provider, tts_provider, llm_provider fields already accept any string. Just pass "huggingface" from the frontend/config.

  4. shared/agent_config.json Add a test preset:

    json
    {
  "id": "preset_huggingface_test",
  "name": "HuggingFace Test",
  "description": "Testing with HuggingFace open-source models",
  "cost_per_min": 1,
  "show": false,
  "llm_provider": "huggingface",
  "llm_model": "meta-llama/Llama-3.3-70B-Instruct",
  "tts_provider": "huggingface",
  "tts_model": "facebook/mms-tts-hin",
  "stt_provider": "huggingface",
  "stt_model": "openai/whisper-large-v3-turbo",
  "supported_languages": ["hi", "en"],
  "default_tts_language": "hi",
  "default_stt_language": "hi"
}

Code Quality Requirements ​

Follow Existing Patterns ​

  • STT/TTS providers MUST extend BaseSTT/BaseTTS with from_meta() + build() pattern
  • LLM provider follows the constructor + build() pattern
  • All providers register in their respective *_REGISTRY dicts
  • Use setup_logger("huggingface_xxx") for all logging

Error Handling ​

  • Handle HuggingFace API rate limits (HTTP 429) with exponential backoff
  • Handle model loading delays (HF serverless models can take 20-60s to cold start HTTP 503 with estimated_time in response)
  • Handle audio format conversion errors gracefully
  • Log all API errors with provider name, model name, and HTTP status
  • Never crash the agent session fall back to default provider if HF fails

Production Readiness ​

  • API key from env var only (HUGGINGFACE_API_KEY), never hardcoded
  • Timeout configuration: HF API can be slow on cold starts. Set reasonable timeouts (30s for first call, 10s for subsequent)
  • Add X-Wait-For-Model: true header for serverless API calls this tells HF to wait for model to load instead of returning 503
  • Audio conversion: Use wave stdlib module for wav encoding/decoding. Avoid heavy dependencies

File Structure ​

Agent/src/providers/
├── llm_providers.py          # Add HuggingFaceLLM class here
├── stt_providers.py          # Add HuggingFaceSTT to registry here
├── tts_providers.py          # Add HuggingFaceTTS to registry here
└── custom/
    ├── __init__.py
    ├── huggingface_stt.py    # Custom STT implementation
    └── huggingface_tts.py    # Custom TTS implementation

Testing Plan ​

Phase 1: LLM Only (Day 1) ​

  • Implement HuggingFaceLLM in llm_providers.py
  • Test with existing Deepgram STT + Cartesia TTS (only swap the LLM)
  • Verify Hindi conversation quality with Llama 3.3 70B
  • Compare response latency vs current Google Gemini

Phase 2: STT (Day 2-3) ​

  • Implement HuggingFaceSTT custom plugin
  • Test with Whisper large-v3-turbo for Hindi transcription
  • Compare accuracy vs Deepgram nova-3 on Hindi phone calls
  • Measure latency impact (non-streaming vs Deepgram streaming)

Phase 3: TTS (Day 3-4) ​

  • Implement HuggingFaceTTS custom plugin
  • Test with facebook/mms-tts-hin for Hindi speech
  • Compare voice quality vs Cartesia sonic-3
  • Measure latency impact (non-streaming vs Cartesia streaming)

Phase 4: Full Pipeline Test (Day 5) ​

  • Run full HuggingFace pipeline: HF STT → HF LLM → HF TTS
  • Make test phone calls and evaluate end-to-end quality
  • Document latency, quality, and cost comparison
ResourceURL
HuggingFace Inference API docshttps://huggingface.co/docs/api-inference/
HuggingFace Inference Endpointshttps://huggingface.co/docs/inference-endpoints/
HF OpenAI-compatible Messages APIhttps://huggingface.co/docs/text-generation-inference/messages_api
HuggingFace Hub Python libraryhttps://huggingface.co/docs/huggingface_hub/
LiveKit custom STT docshttps://docs.livekit.io/agents/v1/integrations/stt/
LiveKit custom TTS docshttps://docs.livekit.io/agents/integrations/tts/
LiveKit OpenAI plugin (base_url)https://docs.livekit.io/python/livekit/plugins/openai/index.html
LiveKit all plugins listhttps://docs.livekit.io/reference/python/livekit/plugins/index.html
Hindi STT models on HFhttps://huggingface.co/models?language=hi&pipeline_tag=automatic-speech-recognition
Hindi TTS models on HFhttps://huggingface.co/models?language=hi&pipeline_tag=text-to-speech

Acceptance Criteria ​

  • [ ] huggingface works as llm_provider in agent config uses HF Inference API via OpenAI-compatible endpoint
  • [ ] huggingface works as stt_provider custom STT class handles audio → text via HF API
  • [ ] huggingface works as tts_provider custom TTS class handles text → audio via HF API
  • [ ] Hindi language works for all three (STT, TTS, LLM)
  • [ ] Proper error handling for rate limits, cold starts, and API failures
  • [ ] Falls back gracefully if HF API is unavailable
  • [ ] HUGGINGFACE_API_KEY env var documented in .env.example
  • [ ] Test preset added to agent_config.json
  • [ ] Code follows existing provider registry pattern
  • [ ] Logging matches existing format using setup_logger()

Last updated: