Concurrent Calling System Developer Guide

What This Doc Covers

How Oktivo controls how many calls happen at the same time. This includes the 4-layer concurrency hierarchy, provider capacity limiting, circuit breaker, TRAI auto-pause/resume, ETA estimation, and how multiple campaigns share slots.

---

Redis Commands Used (Quick Reference)

Before diving in, here's what the Redis commands mean in plain English:

Command	What It Does	Example
SADD key member	Add a member to a Set (like adding to a list, but no duplicates)	SADD org:org123:active_calls 9876543210 → adds phone to org's active set
SREM key member	Remove a member from a Set	SREM org:org123:active_calls 9876543210 → removes phone when call ends
SCARD key	Count how many members are in a Set (O(1), instant)	SCARD org:org123:active_calls → returns 3 (3 active calls)
SMEMBERS key	Get all members of a Set	SMEMBERS provider:tts:cartesia:active → returns all active phones
SISMEMBER key member	Check if a member exists in a Set (true/false)	SISMEMBER provider:tts:cartesia:active 9876543210 → 1 (yes) or 0 (no)
HSET key field value	Set a field in a Hash (like a dictionary/object)	HSET campaign:abc:active 9876543210 '{"name":"John",...}'
HDEL key field	Delete a field from a Hash	HDEL campaign:abc:active 9876543210
HLEN key	Count fields in a Hash	HLEN campaign:abc:active → 5 (5 active calls in this campaign)
SET key value EX seconds	Store a string with auto-expiry	SET cb:tts:cartesia:tripped "1" EX 60 → auto-deletes after 60s
EXISTS key	Check if a key exists (1=yes, 0=no)	EXISTS cb:tts:cartesia:tripped → 1 means circuit breaker is tripped
ZADD key score member	Add to a Sorted Set with a score (used for time-based data)	ZADD cb:tts:cartesia:failures 1711900000000 "uniqueId"
ZCOUNT key min max	Count members in a Sorted Set within a score range	ZCOUNT cb:tts:cartesia:failures 1711899970000 +inf → failures in last 30s
ZREMRANGEBYSCORE key min max	Remove members outside a score range	Prunes old failure entries
SCAN cursor MATCH pattern	Iterate keys matching a pattern (safe for production, unlike KEYS)	SCAN 0 MATCH call:abc:*:providers
DEL key	Delete a key	DEL call:abc:9876543210:providers

Why Sets? Sets give us: no duplicates (same phone can't be counted twice), O(1) add/remove/count, and the ability to inspect which phones are active. Perfect for tracking concurrent calls.

---

Quick Overview

Every 2 seconds, fillCallSlots() runs for each active campaign via BullMQ. It figures out how many new calls to dispatch by computing the minimum of all concurrency constraints:

effectiveSlots = min(campaignLimit, customerLimit, orgLimit, ...providerLimits)

If any constraint says "0 slots available", no calls go out.

---

The 4-Layer Concurrency Hierarchy

Layer 1: Campaign Limit

• Set per campaign via maxConcurrentCalls field on the Campaign model
• Stored in MongoDB Campaign collection
• If not set, falls through to customer/org/platform default
• This is the max for THIS campaign only

Layer 2: Customer Limit (SHARED across all campaigns)

• Set by org admin on the User model (maxConcurrentCalls)
• Limits total concurrent calls across ALL campaigns for that customer
• Tracked via Redis set: customer:{customerUID}:active_calls
• Example: Customer has limit=2 and starts 5 campaigns → only 2 calls active at any time across all 5

Layer 3: Organization Limit (SHARED across all customers)

• Set by SuperAdmin in Organization.plan.maxConcurrentCalls
• Limits total concurrent calls across ALL customers in the org
• Tracked via Redis set: org:{orgUID}:active_calls
• Example: Org has limit=5, Customer A uses 2, Customer B uses 3 → org is at 5/5, nobody gets more

Layer 4: Provider Capacity Limits (SHARED across ALL orgs)

• Set by SuperAdmin in ProviderCapacity MongoDB collection
• Limits based on actual external provider capacity (TTS, STT, SIP, LLM)
• Tracked via Redis sets: provider:{type}:{name}:active
• Cross-org a single TTS provider set contains phones from ALL orgs on the platform

---

How Multiple Campaigns Share Slots

This is important to understand. When a customer starts 5 campaigns with a 2 concurrent call limit:

All 5 campaigns are technically "running" (BullMQ jobs active), but they take turns sharing the 2 slots. The UI shows "Waiting for call slots" on campaigns that are running but have 0 active calls.

---

How fillCallSlots() Works Step by Step

What happens at each step:

1. Lock Only one server instance can fill slots for a campaign at a time (prevents double-dispatch)
2. Campaign check HLEN counts active calls in this campaign's hash. Subtract from campaign max.
3. Org check SCARD counts ALL active phones across ALL customers in the org. If at org limit, return early.
4. Customer check SCARD counts ALL active phones across ALL campaigns for this customer. If at customer limit, return early.
5. Provider check For each provider (TTS, STT, SIP, LLM), check circuit breaker, then compute available = maxConcurrent - reservedBuffer - activeCount. If no ProviderCapacity record exists, treat as unlimited.
6. Final slots Take the minimum of all constraints. If 0, return.
7. Dispatch Fetch pending contacts from MongoDB, dispatch each to LiveKit, track in Redis.

---

Example: Slot Calculation

Setup: Org limit=5, Customer limit=2, Campaign limit=20, TTS max=50 (buffer=5), SIP max=30

Constraint	Max	Active	Available
Campaign limit	20	0	20
Customer limit	2	0	2
Org limit	5	3 (other customers)	2
TTS (cartesia)	50 - 5 buffer = 45	40	5
SIP (telnyx)	30 - 0 buffer = 30	25	5

Effective slots = min(20, 2, 2, 5, 5) = 2

The customer limit is the bottleneck here. Only 2 new calls dispatched.

---

Complete Call Lifecycle (What Happens in Redis)

Why store the provider mapping per call? When a call ends, we need to know which providers it was using. We can't re-resolve the agent preset because the SuperAdmin might have changed it mid-call. So we store the mapping at dispatch time (SET) and read it back at cleanup (GET).

---

Redis Key Layout

Existing Keys (unchanged by our changes)

campaign:{uid}:state          → Hash (isRunning, maxConcurrent, phase, etc.)
campaign:{uid}:active         → Hash (phone → JSON ActiveCallData)
campaign:{uid}:config         → Hash (sipTrunkId, sipPhoneNumber, provider)
campaign:{uid}:lock           → String (distributed lock, 5s TTL)
org:{orgUID}:active_calls     → Set of phone numbers (ALL customers in org)
customer:{uid}:active_calls   → Set of phone numbers (ALL campaigns for customer)
global:active_calls           → Set of phone numbers (ALL calls on platform)

New Keys (added by Provider Capacity Limiter)

provider:{type}:{name}:active              → Set of phone numbers (ALL orgs using this provider)
  Example: provider:tts:cartesia:active    → {"9876543210", "9876543211", "9876543212"}

call:{campaignUID}:{phone}:providers       → String (JSON mapping stored at dispatch, read at cleanup)
  Example: call:abc:9876543210:providers   → '{"tts":"cartesia","stt":"deepgram","llm":"google","sip":"telnyx"}'

cb:{type}:{name}:failures                  → Sorted Set (score=timestamp in ms, member=unique ID)
  Example: cb:tts:cartesia:failures        → used for sliding-window failure counting

cb:{type}:{name}:tripped                   → String "1" with TTL=60s (auto-deletes when cooldown expires)
  Example: cb:tts:cartesia:tripped         → EXISTS returns 1 = provider is tripped

---

Circuit Breaker

Protects against wasting SIP minutes when a provider is failing (e.g., TTS down = callers hear silence).

How it works internally:

1. recordFailure("tts", "cartesia") → ZADD cb:tts:cartesia:failures <timestamp> <uniqueId>
2. Prune old entries: ZREMRANGEBYSCORE cb:tts:cartesia:failures -inf <30s ago>
3. Count recent: ZCOUNT cb:tts:cartesia:failures <30s ago> +inf
4. If count > 5: SET cb:tts:cartesia:tripped "1" EX 60 (trips for 60 seconds)
5. isTripped("tts", "cartesia") → EXISTS cb:tts:cartesia:tripped → 1 = tripped, 0 = healthy

When tripped, fillCallSlots() treats that provider as having 0 available capacity → no calls dispatched using it.

---

TRAI Auto-Pause / Resume

Indian regulation: outbound calls only between 9:00 AM and 9:00 PM IST.

Note: The default 9 AM - 9 PM window can be customized per user via the Dynamic Call Window feature. See DYNAMIC-CALL-WINDOW.md for details.

Key behaviors:

• Auto-stop does NOT mark pending contacts as failed (unlike manual stop)
• If user manually stops during auto-stopped state → autoStoppedByWindow set to false → campaign won't auto-resume next morning
• If startCallingProcess fails during resume → autoStoppedByWindow set back to true → retries next minute

---

ETA Calculation

Pure function in eta.service.ts, computed on-demand (never stored in DB).

totalCallTime = totalContacts × avgCallDuration (default 60s)
parallelTime  = totalCallTime / effectiveConcurrency
durationHours = parallelTime / 3600
creditCost    = totalContacts × (avgDuration / 60) × costPerMinute

effectiveConcurrency = min(campaign limit, customer limit, org limit, provider limits). Falls back to platform default (10) if no limits are set.

Edge cases:

• totalContacts = 0 → all fields return 0
• effectiveConcurrency ≤ 0 → duration = Infinity
• No provider capacity records → uses campaign/customer/org limits only

---

All Database Changes

MongoDB New Collection

ProviderCapacity (src/models/ProviderCapacity.ts)

{
  providerType: "tts" | "stt" | "sip" | "llm",   // which type of provider
  providerName: "cartesia",                        // provider name (lowercase, trimmed)
  maxConcurrent: 50,                               // max concurrent calls this provider supports
  reservedBuffer: 5,                                // reserved slots (safety margin)
  isActive: true,                                   // false = treated as 0 capacity
  notes: "Pro plan",                                // optional notes
  createdAt, updatedAt                              // timestamps
}
Indexes:
  - { providerType: 1, providerName: 1 } UNIQUE    // no duplicate type+name combos
  - { providerType: 1 }                            // for queries

MongoDB Modified Collection

Campaign (src/models/Campaign.ts) added 1 field:

autoStoppedByWindow: Boolean (default: false)

New index: { autoStoppedByWindow: 1, isRunning: 1 } used by the TRAI auto-resume query.

TypeScript Types New Interfaces

Added to src/types/index.ts:

• IProviderCapacity matches the MongoDB schema above
• ResolvedProviders { tts: string, stt: string, llm: string, sip: string }
• ETAInput { totalContacts, effectiveConcurrency, averageCallDurationSeconds?, costPerMinute? }
• ETAResult { estimatedDurationHours, estimatedCalendarDays, estimatedCreditCost }
• Extended ICampaign with autoStoppedByWindow?: boolean
• Extended ActiveCallData with providers?: ResolvedProviders

---

All File Changes Complete List

Backend New Files

File	Purpose
src/models/ProviderCapacity.ts	MongoDB model for provider capacity registry
src/services/providerTracker.service.ts	Redis-backed provider active call tracking (SADD/SREM/SCARD)
src/services/circuitBreaker.service.ts	Sliding-window failure counter + trip state
src/services/eta.service.ts	calculateETA() pure function + getEffectiveConcurrency()
src/controllers/providerCapacity.controller.ts	SuperAdmin CRUD + real-time dashboard API

Backend Modified Files

File	What Changed
src/types/index.ts	Added 4 new interfaces, extended ICampaign and ActiveCallData
src/models/Campaign.ts	Added autoStoppedByWindow field + compound index
src/services/calling.service.ts	addToActiveCalls → tracks providers; removeFromActiveCalls → cleans up providers; fillCallSlots → provider capacity check block; cleanupCampaignState → cleans up provider tracking; exported cleanupCampaignState and getCampaignQueue
src/services/scheduler.service.ts	Added autoStopCampaignsAtWindowEnd() and autoResumeCampaignsAtWindowStart() cron jobs
src/controllers/campaign.controller.ts	Upload response includes ETA/TRAI/credit; new getCampaignETA handler; listCampaigns includes autoStoppedByWindow; manual stop sets autoStoppedByWindow=false
src/routes/superadmin.routes.ts	Added 5 provider capacity routes under SuperAdmin auth
src/routes/campaign.routes.ts	Added GET /:campaignUID/eta route

Frontend New Files

File	Purpose
src/services/providerCapacity.service.ts	API client for provider capacity CRUD + dashboard + campaign ETA
src/pages/superadmin/ProviderCapacityPage.jsx	Provider capacity dashboard with CRUD, real-time utilization, warnings

Frontend Modified Files

File	What Changed
src/pages/superadmin/SuperAdminDashboard.jsx	Added "Provider Capacity" quick link card
src/layouts/SuperAdminLayout.jsx	Added "Provider Capacity" sidebar nav item
src/App.jsx	Added /superadmin/provider-capacity route
src/pages/UploadPage.jsx	Collapsible agent details; ETA/TRAI/credit display after upload; removed auto tab-switch
src/pages/CampaignsPage.jsx	TRAI notice banner; draft campaign ETA estimates; running campaign ETA with "waiting for slots" state; auto-stopped badge
src/MainLayout.jsx	Added autoStoppedByWindow to campaign transform

---

API Endpoints

SuperAdmin Provider Capacity (requires superadmin role)

GET    /api/superadmin/provider-capacity                    → List all records (sorted by type, name)
GET    /api/superadmin/provider-capacity/available-providers → List providers from registry/trunks
GET    /api/superadmin/provider-capacity/dashboard          → Real-time utilization + org warnings
POST   /api/superadmin/provider-capacity                    → Create record (409 on duplicate)
PUT    /api/superadmin/provider-capacity/:id                → Update record (validates constraints)
DELETE /api/superadmin/provider-capacity/:id                → Delete record

Available Providers Endpoint:

• Returns unique provider names from ProviderModelRegistry (for TTS/STT/LLM) and SIPTrunk (for SIP)
• Indicates which providers already have capacity configured (hasCapacity: true)
• Used by the UI to auto-populate the provider dropdown when adding capacity

Customer Campaign ETA (requires auth)

GET    /api/v2/campaigns/:campaignUID/eta           → Current ETA with contact progress
POST   /api/v2/campaigns/upload-campaign            → Response now includes eta, traiWindow, creditCheck

---

Backward Compatibility

• If NO ProviderCapacity records exist → system behaves exactly like before (campaign/customer/org limits only)
• All existing Redis keys are preserved alongside new provider keys
• Provider capacity check is wrapped in try/catch if it fails, falls back to existing limits
• autoStoppedByWindow defaults to false on all existing campaigns no migration needed
• Circuit breaker defaults to "healthy" if Redis is unavailable
• ETA computation is non-fatal upload succeeds even if ETA fails