Scaling Code Changes Developer Changelog

Why These Changes Were Made

We're preparing to handle 10 Lakh (1 Million) calls/month with ~200 peak concurrent calls. The old architecture stored all campaign state in a JavaScript Map inside a single Node.js process. This meant:

• No horizontal scaling running 2+ backend instances would cause each to have its own separate state, leading to duplicate calls and race conditions
• State lost on crash/restart if the process dies, all running campaign state is gone
• No org-level rate limiting one org could consume all system resources
• Hardcoded concurrency maxConcurrent = 2 was baked into the code

These changes move us to a Redis-backed, distributed architecture where multiple backend instances can coordinate safely.

---

What Changed (File by File)

1. config/redis.ts (NEW FILE)

Redis client singleton using ioredis. All campaign state, job queues, and distributed locks go through this.

• getRedis() returns singleton Redis client (configured with maxRetriesPerRequest: null as required by BullMQ)
• closeRedis() graceful shutdown

Why ioredis? BullMQ requires it. It's also faster than the redis npm package and supports pipelining, Lua scripts, and cluster mode out of the box.

2. config/index.ts

Added redisUrl field reading from REDIS_URL env var (defaults to redis://localhost:6379).

3. config/database.ts

MongoDB connection pool upgraded:

• maxPoolSize: 50 (was default 5)
• minPoolSize: 5
• maxIdleTimeMS: 30000
• serverSelectionTimeoutMS: 5000
• socketTimeoutMS: 45000

Why? At 200 concurrent calls, each triggering multiple DB operations (status updates, webhook processing, credit checks), the default pool of 5 connections would exhaust instantly and requests would queue up.

4. models/Campaign.ts

Added maxConcurrentCalls field:

maxConcurrentCalls: { type: Number, default: null, min: 1, max: 500 }

When null, falls back to the org's plan.maxConcurrentCalls. This lets admins set per-campaign limits.

5. types/index.ts

• ICampaign added maxConcurrentCalls?: number | null
• CampaignState completely redesigned for Redis serialization:
  • Removed Map fields, removed setInterval references
  • All fields are primitive types (string, number, boolean) that serialize cleanly to Redis hashes
• ActiveCallData.startTime changed from Date to string (ISO format) for Redis serialization

6. services/calling.service.ts (MAJOR REWRITE)

This is the core change. The entire campaign orchestration engine was rewritten.

Redis Key Layout

campaign:{uid}:state   → Hash  (isRunning, maxConcurrent, phase, userId, AgentID, ...)
campaign:{uid}:active  → Hash  (phone → JSON ActiveCallData)
campaign:{uid}:config  → Hash  (sipTrunkId, sipPhoneNumber, provider, trunkUID)
campaign:{uid}:lock    → String (distributed lock for fillCallSlots, TTL 5s)
org:{orgUID}:active    → Set   (all active phone numbers for this org)
global:active_calls    → Set   (all active phone numbers system-wide)

What Replaced What

Before (in-memory)	After (Redis-backed)
const campaignStates = new Map()	HSET/HGETALL campaign:{uid}:state
state.activeCalls (Map)	HSET/HDEL campaign:{uid}:active
setInterval(manageCampaignCalls, 2000)	BullMQ repeatable job every 2s
clearInterval(state.interval)	queue.removeRepeatableByKey(...)
No distributed locking	SET NX EX 5 lock per campaign in fillCallSlots
No org-level limiting	SCARD org:{orgUID}:active_calls checked before dispatching
maxConcurrent = 2 hardcoded	Resolved: campaign → org.plan → platform default (10)

BullMQ Campaign Orchestrator

Instead of setInterval, we now use a BullMQ queue called campaign-orchestrator:

• When a campaign starts → a repeatable job is added (runs every 2s)
• BullMQ distributes jobs across all backend instances automatically
• When a campaign stops → the repeatable job is removed
• Worker concurrency: 5 (processes 5 campaign ticks in parallel)
• Rate limiter: max 50 jobs/second

Benefit: If you run 3 backend instances, BullMQ ensures each campaign's management loop runs on exactly one instance at a time. No duplicate calls, no race conditions.

Distributed Lock in fillCallSlots

fillCallSlots is the function that picks pending contacts from MongoDB and dispatches calls. With multiple backend instances, two instances could try to fill slots for the same campaign simultaneously, causing duplicate calls.

Solution: SET campaign:{uid}:lock 1 NX EX 5 only one instance can hold the lock. TTL of 5 seconds ensures the lock auto-releases if the holder crashes.

Org-Level Rate Limiting

Before dispatching calls, fillCallSlots now checks:

1. Campaign-level: activeCount < state.maxConcurrent
2. Org-level: SCARD org:{orgUID}:active_calls < org.plan.maxConcurrentCalls

This prevents one campaign from consuming all of an org's concurrent call slots, and prevents one org from consuming all system resources.

maxConcurrent Resolution Chain

campaign.maxConcurrentCalls  →  org.plan.maxConcurrentCalls  →  10 (platform default)

The first non-null, positive value wins. This gives flexibility:

• SuperAdmin sets org limit (e.g., 100)
• Admin/Customer can set per-campaign limit (e.g., 50 for a specific campaign)
• Platform default (10) is the safety net

Stale Call Cleanup

The management loop now detects calls that have been "active" for more than 5 minutes without any webhook update. These are marked as failed and removed from Redis. This handles edge cases where LiveKit webhooks are lost.

Exports (unchanged signatures)

All existing exports are preserved with the same call signatures:

• startCallingProcess(campaignUID, userId, agentId) no change
• stopCallingProcess(campaignUID) no change
• handleCallStatusUpdate(phone, status, metadata?, campaignId?) no change
• recoverCampaignStates() no change
• getCampaignCallStatus(campaignUID) no change
• debugCampaignStates() no change
• processNextBatchFromDB(campaignUID) no change (now delegates to fillCallSlots)

New exports for server.ts:

• initCampaignWorker() starts the BullMQ worker
• closeCampaignWorker() graceful shutdown

7. server.ts

• Redis connection validated on startup (redis.ping())
• BullMQ campaign worker initialized on startup
• Graceful shutdown now closes: Langfuse → BullMQ worker → Redis → Mongoose

8. app.ts

Health check upgraded from { status: "ok" } to:

{
  "status": "ok",        // or "degraded"
  "checks": {
    "mongodb": "ok",     // or "disconnected" / "error"
    "redis": "ok"        // or "error"
  }
}

Returns HTTP 200 when healthy, 503 when degraded. This is what ALB/ECS health checks will use.

9. docker-compose.dev.yml

Added for local development:

• redis service (redis:7-alpine on port 6379)
• redis-insight service (Redis GUI on port 5540)
• client_backend now depends on redis and gets REDIS_URL=redis://redis:6379

10. .env.example and .env

Added REDIS_URL=redis://localhost:6379.

11. package.json

New dependencies:

• ioredis Redis client
• bullmq distributed job queue built on Redis

---

How to Run Locally

1. Start Redis (either via Docker Compose or standalone):

   # Option A: Docker Compose (recommended)
   docker compose -f docker-compose.dev.yml up redis redis-insight -d
   
   # Option B: Standalone Docker
   docker run -d --name redis -p 6379:6379 redis:7-alpine

2. Make sure REDIS_URL=redis://localhost:6379 is in your client_backend/.env

3. Start the backend as usual. On startup you should see:

   Redis connected
   Redis ping OK
   BullMQ campaign-orchestrator worker started

4. RedisInsight is available at http://localhost:5540 add a connection to redis:6379 (or localhost:6379 if running outside Docker) to inspect campaign state keys in real time.

---

What's Done vs What's Remaining

Done (P0 Blockers)

• [x] Campaign state → Redis migration
• [x] BullMQ replacing setInterval
• [x] Distributed locking for fillCallSlots
• [x] maxConcurrent configurable (campaign → org → platform default)
• [x] Health check endpoint (MongoDB + Redis)
• [x] MongoDB connection pool (50 connections)
• [x] Graceful shutdown (Redis + BullMQ + Mongoose)
• [x] Per-org rate limiting via Redis sets
• [x] Stale call cleanup
• [x] Redis for local dev (docker-compose.dev.yml)

Remaining (P1 Important, next sprint)

• [ ] Webhook processing queue (BullMQ queue for LiveKit webhooks instead of inline processing)
• [ ] Egress type switch (Room Composite → Track Egress for lower CPU usage)
• [ ] API endpoints for SuperAdmin/Admin to set maxConcurrentCalls

Remaining (P2 Nice to have)

• [ ] Structured logging (pino/winston replacing console.log)
• [ ] MongoDB compound index review for high-frequency queries
• [ ] Global system-wide concurrent call cap (currently tracked in Redis but not enforced with a hard limit)

---

Infrastructure Changes (Docker / Nginx / Env)

12. docker-compose.yml (Production)

Added redis service:

• redis:7-alpine with appendonly yes for persistence
• maxmemory 512mb with allkeys-lru eviction as safety cap
• Port bound to 127.0.0.1:6379 only (not exposed to internet)
• Persistent volume redis_data
• client_backend now depends on redis and gets REDIS_URL=redis://redis:6379

This is a temporary setup for the current EC2 deployment. After ECS migration, this Redis container gets replaced by ElastiCache (see DEVOPS-REQUIREMENTS.md).

13. docker-compose.dev.yml

Added:

• redis service on port 6379
• redis-insight GUI on port 5540 for debugging
• client_backend depends on redis, gets REDIS_URL=redis://redis:6379
• volumes: redis_data for persistence

14. nginx.conf

Added production-ready settings:

• proxy_read_timeout 120s prevents nginx from killing long webhook processing
• proxy_send_timeout 120s same for response
• proxy_connect_timeout 10s fast fail on backend down
• client_max_body_size 50m allows large CSV uploads and webhook payloads

No structural changes needed nginx will be replaced by ALB after ECS migration.

15. .env / .env.example files

• client_backend/.env added REDIS_URL=redis://localhost:6379
• client_backend/.env.example added REDIS_URL=redis://localhost:6379
• livekit_selfhosted/.env.example updated webhook URL comment with correct stage/prod paths

16. No changes needed

These files were reviewed and confirmed correct as-is:

• client_backend/Dockerfile already does npm ci which picks up new ioredis/bullmq deps from package-lock.json
• .github/workflows/deploy.yml copies docker-compose.yml to EC2 and runs docker compose up -d, which will automatically start the new Redis service
• livekit_selfhosted/docker-compose.yml LiveKit's own Redis is separate from backend Redis, no changes needed