Concurrency Management System Design & Implementation Guide

Overview

This document describes the comprehensive concurrency management system for Oktivo Voice Platform. The system enables hierarchical quota allocation (SuperAdmin → Organization → Customer) with real-time monitoring across TTS, STT, LLM, and Channel (SIP) providers.

Implementation Status: ✅ COMPLETE

Changes made:

• Backend types, models, services, controllers, and routes
• Frontend types, services, and SuperAdmin concurrency dashboard
• Removed reservedBuffer from provider capacity (as per user request)
• Added sidebar link to new Concurrency Management page

Problem Statement

Current State

1. SuperAdmin can set provider capacity limits via ProviderCapacity collection
2. Org/Customer concurrency limits exist but are disconnected from provider capacities
3. No visibility into which provider is the bottleneck
4. No warning when allocated quotas exceed provider capacity
5. Customers may expect 10 concurrent calls but only get 5 due to provider limits

Key Insight

Concurrency quotas are only meaningful within the context of a specific preset's provider stack.

A "10 concurrent call quota" means different things depending on which preset is used:

• Preset A (Cartesia/Deepgram/OpenAI/Telnyx): Effective = min(100, 150, 200, 80) = 80 → 10 is achievable
• Preset B (Deepgram TTS/Whisper/Claude/Plivo): Effective = min(5, 50, 100, 30) = 5 → only 5 achievable

Architecture

Data Model

1. ProviderCapacity Collection (Simplified)

interface IProviderCapacity {
  providerType: "tts" | "stt" | "sip" | "llm";
  providerName: string;         // e.g., "cartesia", "deepgram", "openai"
  maxConcurrent: number;        // Total concurrent calls for this provider
  isActive: boolean;
  notes?: string;
  // REMOVED: reservedBuffer (per user request)
  createdAt: Date;
  updatedAt: Date;
}

2. Organization Model (Extended)

// New fields added to Organization schema
{
  concurrencyQuota: {
    allocated: number;          // Assigned by SuperAdmin
    allocatedAt: Date | null;
    allocatedByUID: string | null;
  }
}

3. User Model (Extended)

// New fields added to User schema
{
  maxConcurrentCalls: number;   // Already exists
  concurrencyQuotaAllocatedAt: Date | null;
  concurrencyQuotaAllocatedByUID: string | null;
}

Redis Key Structure

# Existing keys (unchanged)
provider:{type}:{name}:active              → Set (phone numbers across ALL orgs)
campaign:{uid}:state                       → Hash
campaign:{uid}:active                      → Hash (phone → JSON)
org:{orgUID}:active_calls                  → Set (phone numbers)
customer:{customerUID}:active_calls        → Set (phone numbers)

# New keys for detailed monitoring
call:{roomName}:metadata                   → Hash (orgUID, customerUID, campaignUID, agentUID, providers JSON, startTime)

Concurrency Hierarchy

┌─────────────────────────────────────────────────────────────────────────┐
│                    Platform Provider Capacity                            │
│  TTS: min(cartesia:100, elevenlabs:50, ...) = platform_tts_capacity     │
│  STT: min(deepgram:150, whisper:50, ...) = platform_stt_capacity        │
│  LLM: min(openai:200, anthropic:100, ...) = platform_llm_capacity       │
│  SIP: min(telnyx:80, plivo:50, ...) = platform_sip_capacity             │
│  Platform Effective = min(all provider capacities)                       │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  Organization Quota (allocated by SuperAdmin)                            │
│  org.concurrencyQuota.allocated ≤ platform effective                    │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  Customer Quota (allocated by Org Admin)                                 │
│  user.maxConcurrentCalls ≤ org.concurrencyQuota.available               │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  Campaign Limit                                                          │
│  campaign.maxConcurrentCalls ≤ user.maxConcurrentCalls                  │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  Runtime Effective Concurrency                                           │
│  effectiveSlots = min(campaign, customer, org, ...presetProviderLimits) │
└─────────────────────────────────────────────────────────────────────────┘

Preset Capacity Calculation

Each preset uses specific providers. The preset's effective capacity is:

presetEffectiveCapacity = min(
  ProviderCapacity[preset.tts_provider],
  ProviderCapacity[preset.stt_provider],
  ProviderCapacity[preset.llm_provider],
  ProviderCapacity[preset.sip_provider]   // from campaign/trunk config
)

API Endpoints

SuperAdmin APIs

Provider Capacity (Enhanced)

GET    /api/superadmin/provider-capacity                    → List all capacity records
GET    /api/superadmin/provider-capacity/available-providers → List providers from registry/trunks
GET    /api/superadmin/provider-capacity/dashboard          → Real-time utilization dashboard
POST   /api/superadmin/provider-capacity                    → Create capacity record
PUT    /api/superadmin/provider-capacity/:id                → Update capacity record
DELETE /api/superadmin/provider-capacity/:id                → Delete capacity record

Available Providers (NEW)

GET /api/superadmin/provider-capacity/available-providers
Response: {
  success: true,
  providers: {
    tts: [{ name: "cartesia", hasCapacity: false }, { name: "elevenlabs", hasCapacity: true }],
    stt: [{ name: "deepgram", hasCapacity: false }],
    llm: [{ name: "openai", hasCapacity: true }, { name: "anthropic", hasCapacity: false }],
    sip: [{ name: "telnyx", hasCapacity: false }, { name: "plivo", hasCapacity: true }]
  }
}

• TTS/STT/LLM providers are fetched from ProviderModelRegistry (active providers only)
• SIP providers are fetched from SIPTrunk collection (active trunks only)
• hasCapacity: true means a capacity record already exists for that provider

Concurrency Dashboard (NEW)

GET /api/superadmin/concurrency/dashboard
Response: {
  success: true,
  platformConcurrency: {
    tts: { total: 200, active: 45, available: 155, providers: [{...}] },
    stt: { total: 150, active: 45, available: 105, providers: [{...}] },
    llm: { total: 300, active: 45, available: 255, providers: [{...}] },
    sip: { total: 100, active: 45, available: 55, providers: [{...}] },
    effective: { total: 100, active: 45, available: 55, bottleneck: "sip" }
  },
  quotaAllocation: {
    platformEffective: 100,
    totalAllocatedToOrgs: 75,
    remaining: 25
  },
  activeCallsByOrg: [{
    orgUID, orgName, quota, active, customers: [{...}]
  }]
}

Preset Capacity Check (NEW)

GET /api/superadmin/presets/capacity
Response: {
  success: true,
  presets: [{
    presetId: "preset_pro_hd",
    presetName: "Pro HD",
    providers: {
      tts: { provider: "cartesia", capacity: 100 },
      stt: { provider: "deepgram", capacity: 150 },
      llm: { provider: "openai", capacity: 200 }
    },
    effectiveCapacity: 100
  }]
}

Organization Quota Allocation (NEW)

POST /api/superadmin/organizations/:orgUID/concurrency-quota
Body: { quota: 50, force: false }
Response: {
  success: true,
  warnings: [{
    presetId, presetName, maxCapacity, reason
  }],
  assigned: true/false,
  organization: {...}
}

Organization Admin APIs

Concurrency Dashboard (NEW)

GET /api/org/concurrency/dashboard
Response: {
  success: true,
  orgQuota: {
    allocated: 50,
    distributed: 30,
    available: 20,
    active: 15
  },
  presetCapacities: [{
    presetId, presetName, effectiveCapacity, providers: {...}
  }],
  customerUsage: [{
    userUID, name, email, quota, active,
    activeCalls: [{ roomName, phone, campaignName, agentName, startTime, duration }]
  }]
}

Customer Quota Assignment (Enhanced)

PUT /api/org/users/:userUID
Body: { maxConcurrentCalls: 10 }
Response: {
  success: true,
  warnings: [{
    presetId, presetName, maxCapacity, reason
  }],
  user: {...}
}

Customer APIs

Campaign Effective Concurrency (NEW)

GET /api/v2/campaigns/:campaignUID/effective-concurrency
Response: {
  success: true,
  campaignUID: "camp_123",
  agentPreset: "preset_budget",
  quotas: {
    campaign: 20,
    customer: 10,
    organization: 50
  },
  providerCapacities: {
    tts: { provider: "deepgram", maxConcurrent: 5, currentActive: 2 },
    stt: { provider: "whisper", maxConcurrent: 50, currentActive: 2 },
    llm: { provider: "openai", maxConcurrent: 200, currentActive: 2 }
  },
  effectiveConcurrency: 5,
  bottleneck: "TTS:deepgram",
  warning: "Your TTS provider (deepgram) limits this campaign to 5 concurrent calls."
}

Frontend Pages

1. SuperAdmin Concurrency Dashboard

Location: /superadmin/concurrency

Features:

• Platform-wide provider capacity overview (TTS, STT, LLM, SIP)
• Real-time active call counts per provider
• Platform effective concurrency display with bottleneck indicator
• Partial configuration warning when some provider types have no capacity configured
• Organization quota allocation summary
• Active calls by organization with drill-down
• Warnings for over-allocated organizations
• Link to Provider Capacity page for configuration

2. Provider Capacity Management

Location: /superadmin/provider-capacity

Features:

• Auto-populated provider dropdown from Provider Registry (TTS/STT/LLM) and SIP Trunks (SIP)
• Real-time utilization dashboard with usage progress bars
• CRUD operations for capacity records
• Circuit breaker status display
• Consuming organizations breakdown
• Organization warnings when quota exceeds capacity
• Empty state with guidance when no providers configured

3. Organization Management Enhancement

Location: /superadmin/org-management

Add to org edit dialog:

• Concurrency quota allocation field
• Preset capacity compatibility warnings
• Show which presets can support the requested quota

4. Org Admin Concurrency Dashboard (NEW)

Location: /admin/concurrency or section in existing dashboard

Features:

• Organization quota summary
• Customer allocation list
• Active calls per customer
• Preset capacity warnings when assigning quotas

5. Customer Campaign Enhancement

Location: Campaign start dialog

Features:

• Show effective concurrency for selected preset
• Display bottleneck provider
• Warning if effective < requested

Implementation Files

Backend New Files

File	Purpose
src/services/concurrency.service.ts	Core concurrency calculation and quota management
src/controllers/concurrency.controller.ts	API handlers for concurrency endpoints
src/routes/concurrency.routes.ts	Route definitions

Backend Modified Files

File	Changes
src/models/ProviderCapacity.ts	Remove reservedBuffer field
src/models/Organization.ts	Add concurrencyQuota embedded document
src/models/User.ts	Add quota metadata fields
src/services/calling.service.ts	Update fillCallSlots() to remove reservedBuffer
src/services/providerTracker.service.ts	Add call metadata storage
src/controllers/providerCapacity.controller.ts	Add getAvailableProviders endpoint
src/routes/superadmin.routes.ts	Add concurrency routes + available-providers route
src/routes/admin.routes.ts	Add org concurrency routes
src/types/index.ts	Add new interfaces
src/app.ts	Mount new routes

Frontend New Files

File	Purpose
src/pages/superadmin/ConcurrencyDashboardPage.tsx	SuperAdmin concurrency dashboard
src/pages/admin/ConcurrencyDashboardSection.tsx	Org admin concurrency section
src/services/concurrency.service.ts	API client for concurrency endpoints
src/components/ui/alert.tsx	Alert component for warnings/notices
src/types/concurrency.ts	TypeScript interfaces

Frontend Modified Files

File	Changes
src/pages/superadmin/ProviderCapacityPage.tsx	Auto-populated provider dropdown, improved UX
src/services/providerCapacity.service.ts	Add getAvailableProviders() API call
src/pages/superadmin/OrgManagementPage.tsx	Add quota allocation with warnings
src/layouts/SuperAdminLayout.tsx	Add sidebar link to concurrency dashboard
src/routes.tsx	Add new routes

Migration Strategy

Phase 1: Backend Preparation

1. Remove reservedBuffer from ProviderCapacity schema
2. Add concurrency quota fields to Organization/User schemas
3. Create ConcurrencyService
4. Update fillCallSlots() to not use reservedBuffer

Phase 2: API Implementation

1. Create concurrency dashboard endpoints
2. Create quota allocation endpoints
3. Create preset capacity check endpoints
4. Add warnings to org/user update endpoints

Phase 3: Frontend Implementation

1. Create SuperAdmin concurrency dashboard
2. Add capacity tab to Provider Registry
3. Enhance Org Management with quota allocation
4. Create Org Admin concurrency section
5. Enhance campaign start dialog

Phase 4: Testing & Validation

1. Unit tests for ConcurrencyService
2. Integration tests for quota allocation
3. E2E tests for dashboard functionality
4. Load testing for real-time updates

Edge Cases

Scenario	Behavior
Provider has no capacity record	Treated as unlimited (log warning)
Org quota > Platform effective	Validation error, allow with force flag and warning
Customer quota > Org available	Validation error
Active calls exceed new quota	Existing calls continue, no new dispatches
Provider deactivated	Existing calls continue, no new calls
Preset uses unregistered provider	Fall back to unlimited with warning
Redis failure	Fall back to MongoDB-only limits

Monitoring & Alerts

Metrics to Track

• Provider utilization percentage per type
• Quota utilization per org/customer
• Effective concurrency vs requested
• Bottleneck provider frequency

Suggested Alerts

• Provider > 80% capacity
• Org quota utilization > 90%
• Multiple campaigns hitting provider limits
• Circuit breaker trips

---

Quick Start

1. Run migration to remove reservedBuffer: npm run migrate:provider-capacity
2. Set provider capacities via SuperAdmin UI
3. Allocate quotas to organizations
4. Org admins distribute quotas to customers
5. Customers see effective concurrency when starting campaigns