Oktivo Testing Guide Step by Step

This guide walks you through every test tool we have, in order, with exact commands and what to expect.

---

Prerequisites

Before you start, make sure you have:

cd client_backend
npm install

Your .env file should already have MONGO_URI, DB_NAME, and other standard config.

For E2E tests (Steps 3 and 4), you also need these env vars:

E2E_BASE_URL=http://localhost:3000
E2E_EMAIL[email protected]
E2E_PASSWORD=yourpassword
E2E_AGENT_ID=your-agent-id
E2E_PHONE=+919876543210

---

Step 1: Integration Tests (no infra needed)

These test the webhook pipeline logic billing, credits, status transitions by writing directly to MongoDB. No LiveKit, no SIP, no real calls. Runs against a separate _test database.

Run

cd client_backend
npx vitest --run src/tests/webhook-pipeline.test.ts

Expected Output

 ✓ src/tests/webhook-pipeline.test.ts (15 tests | 15 passed) ~60s
   ✓ Single Call Lifecycle
     ✓ should complete full call lifecycle with all data stored
     ✓ should handle rejected call (not answered)
     ✓ should handle not-picked call (timeout)
   ✓ Billing Duration Computation
     ✓ should compute correct duration from callAnsweredAt
     ✓ should fall back to participantJoinedAt when callAnsweredAt is missing
     ✓ should return null for unanswered calls
     ✓ should return null for calls with no join timestamp
     ✓ should ceil fractional seconds
   ✓ Credit Deduction
     ✓ should deduct correct credits for a 3-minute call
     ✓ should round up partial minutes
     ✓ should be idempotent (no double billing on webhook retry)
   ✓ Batch Call Simulation
     ✓ should handle 50 concurrent calls correctly
     ✓ should handle mixed outcomes (answered + rejected + timeout)
   ✓ Edge Cases
     ✓ should handle very short call (< 1 second)
     ✓ should handle test call (no credit deduction)

 Test Files  1 passed (1)
      Tests  15 passed (15)

What to look for

• All 15 tests should pass
• You'll see Mongoose duplicate index warnings ignore those, they're cosmetic
• If any test fails, the error message will tell you exactly which assertion broke

If something fails

• Slug must be lowercase alphanumeric with hyphens → the test slug generator has a bug, check testSlug() function
• E11000 duplicate key error on ServicePricing → Run POST /internal/scripts/fix-service-pricing-indexes to drop legacy indexes (see SERVICE-PRICING.md)
• Connection errors → check your MONGO_URI in .env

---

Step 2: Load Test (synthetic, no real calls)

Creates a fake campaign with N contacts and simulates the full webhook pipeline at high concurrency. Tests throughput and data integrity under load.

Run

cd client_backend

# Quick smoke test
npx tsx src/tests/load-test-pipeline.ts --contacts 50 --concurrency 10

# Realistic load
npx tsx src/tests/load-test-pipeline.ts --contacts 500 --concurrency 50

# Keep data for inspection
npx tsx src/tests/load-test-pipeline.ts --contacts 100 --concurrency 20 --skip-cleanup

Expected Output

============================================================
OKTIVO LOAD TEST PIPELINE
Contacts: 50 | Concurrency: 10
============================================================

Connected to MongoDB
Test data created:
  Org:      org-lt-mmx7b2q4
  Customer: usr-lt-mmx7b2q4
  Campaign: camp-lt-mmx7b2q4

Creating 50 contacts...
  50 contacts created

Running call simulations (concurrency=10)...
  Progress: 50/50 (5.8 contacts/sec)

============================================================
RESULTS
============================================================

--- Throughput ---
  Total time:          8.6s
  Contacts processed:  50/50
  Throughput:          5.8 contacts/sec
  Avg processing time: 1570ms per contact

--- Outcomes ---
  answered            28  (56.0%)
  notPicked            9  (18.0%)
  rejected             5  (10.0%)
  timeout              5  (10.0%)
  mediaFailure         3  (6.0%)

--- Billing ---
  Total billing seconds:  28
  Total billing minutes:  1
  Customer credits used:  84
  Org credits used:       28
  Org profit retained:    56

--- Data Integrity Check ---
  Done:           28
  Failed:         22
  Stuck:          0
  With recording: 50
  With transcript:28
  With analysis:  28
  With billing:   28
  Credit txns:    28

--- VERDICT ---
  ALL CHECKS PASSED

Cleaning up test data...
  Cleaned up
============================================================

What to look for

• Stuck: 0 this MUST be zero. Any stuck contacts means the pipeline has a bug.
• ALL CHECKS PASSED at the bottom
• Outcome distribution should be roughly 60% answered, 15% rejected, 15% not picked, etc.
• Contacts processed should equal total contacts

If something fails

• Stuck > 0 → contacts didn't complete the pipeline, check handleCallStatusUpdate logic
• Billing mismatch → some answered calls weren't billed, check credit service
• E11000 duplicate key → ServicePricing legacy index issue
• Slow throughput → check MongoDB connection pool size

---

Step 3: E2E Single Call (real stack)

Triggers a REAL phone call through the entire production stack. Someone needs to answer the phone.

What you need running

• Backend server (npm run dev or deployed)
• Redis
• LiveKit server
• Agent worker(s)
• Analysis service
• A phone that someone will answer

Setup

Make sure these are in your .env:

E2E_BASE_URL=http://localhost:3000
E2E_EMAIL[email protected]
E2E_PASSWORD=yourpassword
E2E_AGENT_ID=your-agent-id
E2E_PHONE=+919876543210

Run

cd client_backend
npx tsx src/tests/e2e-call-test.ts

Or with overrides:

npx tsx src/tests/e2e-call-test.ts --phone +919876543210 --timeout 180 --skip-s3

What happens

1. Script logs in via API
2. Triggers POST /test-agent
3. Your phone rings pick up and talk for 15-30 seconds, then hang up
4. Script polls the Contact document every 3 seconds
5. After call ends, waits 10 seconds for async data (recording, transcription, analysis)
6. Validates everything and prints a report

Expected Output

======================================================================
OKTIVO E2E CALL TEST
======================================================================

  Server:   http://localhost:3000
  Phone:    +919876543210
  Agent:    your-agent-id
  Timeout:  120s
  S3 check: enabled

[1/5] Connecting to MongoDB...
  Connected

[2/5] Authenticating...
  Logging in as [email protected]...
  Logged in as John Doe (role: customer)

[3/5] Triggering test call...
  ☎️  PHONE WILL RING please answer and talk for 15-30 seconds, then hang up
  Calling +919876543210 with agent your-agent-id...
  Call dispatched → room: room-a1b2c3d4, contactUID: abc123xyz

[4/5] Waiting for call to complete...
  [0s] status=in_progress, callStatus=initiating
  [8s] status=done, callStatus=ended_by_user
  Call completed waiting 10s for async data...

[5/5] Validating pipeline...

======================================================================
E2E CALL TEST REPORT
======================================================================

  Contact UID:    abc123xyz
  Room ID:        room-a1b2c3d4
  Final Status:   done / ended_by_user
  Total Time:     22.3s
  Duration:       18s

--- Checks (17/17 passed) ---

  ✅ Call completed
     status=done, callStatus=ended_by_user
  ✅ participantJoinedAt set
     2025-01-15T10:00:00.000Z
  ✅ participantLeftAt set
     2025-01-15T10:00:18.000Z
  ✅ callAnsweredAt set
     2025-01-15T10:00:07.000Z
  ✅ sipCallId present
     sip-abc123
  ✅ roomId matches
     room-a1b2c3d4
  ✅ LiveKit data stored
     +919876543210
  ✅ billingDuration computed
     18s
  ✅ callDuration matches billingDuration
     callDuration=18, billingDuration=18
  ✅ egressId present
     EG_xyz789
  ✅ s3RecordingKey set
     org123/usr456/test_agent_123/abc123xyz/919876543210_20250115_100000.mp4
  ✅ S3 recording file exists
     org123/usr456/test_agent_123/abc123xyz/919876543210_20250115_100000.mp4
  ✅ recordingUrl generated
     present (presigned URL)
  ✅ Transcription stored
     5 items
  ✅ Transcription has agent + user turns
     agent=true, user=true
  ✅ Analysis stored
     sentiment=positive
  ✅ isTestCall flag set
     isTestCall=true
  ✅ No credit deduction (test call)
     correct no transaction

--- VERDICT ---
  ALL CHECKS PASSED full pipeline working correctly
======================================================================

What to look for

• All checks should show ✅
• billingDuration should be roughly how long you talked (in seconds)
• callAnsweredAt should be set (means recording started at the right time)
• S3 recording file exists confirms the recording actually uploaded
• Transcription stored with agent + user turns confirms the agent heard you
• Analysis stored confirms the analysis service processed the call
• No credit deduction confirms test calls aren't billed

Common issues

• Call completed shows ❌ with status=failed → the phone wasn't answered, or agent crashed. Check agent logs.
• callAnsweredAt missing → /start-recording wasn't called by the agent. Check agent's call_handler.py.
• S3 recording file exists fails → egress didn't upload. Check egress worker logs and S3 permissions.
• Transcription stored missing → agent didn't call /transcription-sync. Check agent's post_call.py.
• Analysis stored missing → analysis service didn't call /analysis-sync. Check analysis service logs.
• recordingUrl missing but S3 file exists → egress_ended webhook didn't fire. Check LiveKit webhook config.

---

Step 4: E2E Campaign Test (real orchestration)

Starts a REAL campaign through the API. Tests BullMQ orchestration, Redis state, concurrent call dispatch, and campaign completion.

What you need

• Everything from Step 3
• An existing campaign with contacts uploaded (small batch, 3-5 contacts recommended)
• People to answer the phones (or just you answering one phone the rest will show as failed/rejected)

Run

cd client_backend
npx tsx src/tests/e2e-campaign-test.ts --campaign <campaignUID> --agent <agentId>

Expected Output

======================================================================
OKTIVO E2E CAMPAIGN TEST
======================================================================

  Server:    http://localhost:3000
  Campaign:  camp_abc123
  Agent:     your-agent-id
  Timeout:   300s

[1/4] Connecting to MongoDB...
  Connected
  Campaign: "My Test Campaign" 5 contacts

[2/4] Authenticating...
  Logged in as John Doe

[3/4] Starting campaign...
  ☎️  PHONES WILL RING answer calls and talk for 15-30 seconds
  Campaign started

[4/4] Monitoring campaign...
  [0s]   running=true  | done=0 failed=0 pending=5 active=0 skipped=0 / 5
  [5s]   running=true  | done=0 failed=0 pending=3 active=2 skipped=0 / 5
  [15s]  running=true  | done=1 failed=1 pending=1 active=2 skipped=0 / 5
  [30s]  running=true  | done=2 failed=2 pending=0 active=1 skipped=0 / 5
  [40s]  running=false | done=2 failed=3 pending=0 active=0 skipped=0 / 5
  Campaign completed waiting 15s for async data...
  Validating...

======================================================================
E2E CAMPAIGN TEST REPORT
======================================================================

  Campaign:     camp_abc123
  Total Time:   55.2s
  Contacts:     5
  Done:         2
  Failed:       3
  Skipped:      0

--- Checks (14/14 passed) ---

  ✅ Campaign stopped running
     isRunning=false
  ✅ No stuck contacts
     all contacts reached terminal state
  ✅ All contacts processed
     done=2 + failed=3 + skipped=0 = 5 / 5
  ✅ Done contacts have participantJoinedAt
     2/2
  ✅ Done contacts have participantLeftAt
     2/2
  ✅ Done contacts have callAnsweredAt
     2/2
  ✅ Done contacts have billingDuration
     2/2
  ✅ Done contacts have sipCallId
     2/2
  ✅ Done contacts have egressId
     2/2
  ✅ Done contacts have recordingUrl
     2/2
  ✅ Done contacts have transcription
     2/2
  ✅ Done contacts have analysis
     2/2
  ✅ Done contacts have credit transactions
     2/2 billed
  ✅ Failed contacts have disconnect reason
     3/3

--- VERDICT ---
  ALL CHECKS PASSED full campaign orchestration working correctly
======================================================================

What to look for

• No stuck contacts must pass. Any stuck contact means BullMQ or webhook pipeline has a bug.
• All contacts processed done + failed + skipped should equal total
• Done contacts should have ALL data (timestamps, recording, transcription, analysis, credits)
• Failed contacts should have disconnect reasons and NO billing

After the test

Run the campaign validator for a deeper check:

npx tsx src/scripts/validate-campaign.ts <campaignUID> --verbose

---

Step 5: Campaign Validator (post-hoc check)

Use this after any test or real campaign to check data integrity.

Run

cd client_backend

# Validate a specific campaign
npx tsx src/scripts/validate-campaign.ts <campaignUID>

# Verbose (show all issues)
npx tsx src/scripts/validate-campaign.ts <campaignUID> --verbose

# All running campaigns
npx tsx src/scripts/validate-campaign.ts --all-running

# 10 most recent campaigns
npx tsx src/scripts/validate-campaign.ts --recent 10

Expected Output

======================================================================
CAMPAIGN: My Test Campaign (camp_abc123)
Status: STOPPED
======================================================================

--- Contact Status Breakdown ---
  done           850  (85.0%)  ###########################################
  failed         150  (15.0%)  ########
  TOTAL         1000

--- Call Status Breakdown ---
  ended_by_user       700
  ended_by_agent      150
  rejected             80
  not picked           50
  media_failure        20

--- Performance Stats ---
  Avg call duration:       45s
  Min / Max duration:      3s / 178s
  Total billing minutes:   638
  Avg attempts per contact:1.2

--- Data Completeness (for 'done' contacts) ---
  Recordings:     848/850 (99.8%) 2 MISSING
  Transcriptions: 850/850 (100.0%)
  Analysis:       850/850 (100.0%)

--- Credit Billing ---
  Credit transactions:     850
  Total credits charged:   1914

--- ERRORS (2) ---
  [ERROR] Missing recording URL
          Contact: John Doe (+919876543210) cnt_abc123
          Duration: 45s, egressId: EG_xyz, s3Key: org/usr/camp/cnt/phone.mp4

--- VERDICT ---
  FAILED 2 error(s), 0 warning(s)
======================================================================

What to look for

• VERDICT: ALL CHECKS PASSED is the goal
• Missing recordings → check egress workers and S3 permissions
• Missing transcriptions → check agent's post_call sync
• Missing credit transactions → check credit service logs
• Stuck in_progress → check for stale calls, webhook delivery issues

---

Quick Reference

Test	Command	Infra Needed	Time
Integration	npx vitest --run src/tests/webhook-pipeline.test.ts	MongoDB only	~60s
Load test	npx tsx src/tests/load-test-pipeline.ts --contacts 500 --concurrency 50	MongoDB only	~30s
E2E call	npx tsx src/tests/e2e-call-test.ts	Full stack + phone	~30s
E2E campaign	npx tsx src/tests/e2e-campaign-test.ts --campaign <uid> --agent <id>	Full stack + phones	~1-5min
Validator	npx tsx src/scripts/validate-campaign.ts <uid>	MongoDB only	~5s

---

Daily Health Check

Run this every morning to verify the full stack is working:

npx tsx src/tests/e2e-call-test.ts

One team member answers the phone, 30 seconds, done. If all checks pass, your entire pipeline is healthy.