Langfuse Observability Oktivo

Track LLM tokens, costs, and call metadata across all 3 services filterable by organization, customer, campaign, and individual call.

Environment Setup
Architecture Overview
Files Changed for Langfuse
What Data Goes to Langfuse
What Contributes to Tokens & Cost
Dashboard Setup (Step by Step)
Drill-Down: Org → Customer → Campaign → Call
Custom Model Pricing (Fixes $0.00)
Filter Cheat Sheet
Adding Observability to New Code
Troubleshooting

Environment Setup

Each service needs these env vars (same Langfuse project keys per environment):

env

# Python services (Agent, Analysis)
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com

# Node.js service (client_backend)
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_BASEURL=https://cloud.langfuse.com

Use separate Langfuse projects per environment (dev/stage/prod) so data never mixes.

To get keys: Langfuse dashboard → Project Settings → API Keys.

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│  client_backend (Node.js)                                   │
│                                                             │
│  calling.service.ts ──► trackCallDispatch()                 │
│  openai.service.ts  ──► trackGeneration() [hard_words]      │
│  webhook.controller ──► trackCallCompletion()               │
│                  │                                          │
│          langfuse.service.ts (Langfuse JS SDK)              │
│          + name cache (org/customer/campaign names)         │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
              Langfuse Cloud
                       ▲
                       │
┌──────────────────────┼──────────────────────────────────────┐
│  Agent (Python)      │                                      │
│                      │                                      │
│  LiveKit Agents ──► OTel TracerProvider ──► Langfuse SDK    │
│  (auto STT/LLM/TTS spans)                                  │
│                                                             │
│  post_call.py ──► call_usage_summary trace                  │
│                   (voice_call_llm generation with tokens)   │
│  telemetry.py ──► Langfuse init + OTel setup                │
└──────────────────────┼──────────────────────────────────────┘
                       │
┌──────────────────────┼──────────────────────────────────────┐
│  Analysis (Python)   │                                      │
│                      │                                      │
│  @observe decorators on orchestrator + openai_service       │
│  openai_service.py ──► @observe(as_type="generation")       │
│                        (auto-captures OpenAI tokens/cost)   │
│  config.py ──► Langfuse client init                         │
└─────────────────────────────────────────────────────────────┘

Files Changed for Langfuse

Below is every file that was added or modified for Langfuse integration.

Agent Service (Python)

File	What was changed
`Agent/src/services/telemetry.py`	New file. Initializes Langfuse SDK + OTel TracerProvider so LiveKit auto-emitted STT/LLM/TTS spans flow to Langfuse. Provides `get_langfuse_client()` and `flush_langfuse()` helpers.
`Agent/agent.py`	Calls `setup_langfuse_telemetry()` at worker startup. Calls `flush_langfuse()` on shutdown. Passes `model_info` dict (stt_model, llm_model, tts_model) to `post_call_event`. Collects usage metrics via `UsageCollector`.
`Agent/src/services/post_call.py`	Rewritten. Sends a `call_usage_summary` trace to Langfuse via the Python SDK with 3 generation observations: `voice_call_llm` (input/output tokens), `voice_call_tts` (characters), `voice_call_stt` (milliseconds). Also passes `room_name` + human-readable names to Analysis service.
`Agent/src/schemas/metadata_schema.py`	Added optional fields: `org_name`, `customer_name`, `customer_email`, `campaign_name` for Langfuse name resolution.
`Agent/src/services/metadata_resolver.py`	Added `_normalize_dispatch_fields()` to map `orgUID`→`organization_id`, `customerUID`→`customer_id` from dispatch metadata.

Analysis Service (Python)

File	What was changed
`Analysis/config.py`	Added `_init_langfuse()` to initialize Langfuse client. Provides `get_langfuse()` helper.
`Analysis/app.py`	Added Langfuse flush on FastAPI shutdown via lifespan handler.
`Analysis/src/services/openai_service.py`	Added `@observe(as_type="generation")` decorator on `structured_json_call()`. This auto-captures OpenAI token usage and cost.
`Analysis/src/orchestration/analysis_orchestrator.py`	Added `@observe(name="conversation_analysis")` decorator. Uses `propagate_attributes()` to tag traces with `user_id` (org UID), `session_id` (room_name), tags (`org:`, `customer:`, `campaign:`, `analysis`), and metadata (all names). Added `_resolve_names()` DB fallback. Accepts `room_name` + name fields from Agent.
`Analysis/src/routes/analysis_routes.py`	Passes `room_name`, `org_name`, `customer_name`, `customer_email`, `campaign_name` through to orchestrator.
`Analysis/src/schemas/analysis_schema.py`	Added fields: `room_name`, `org_name`, `customer_name`, `customer_email`, `campaign_name` to `AnalysisRequest`.

client_backend Service (Node.js)

File	What was changed
`client_backend/src/services/langfuse.service.ts`	New file. Langfuse singleton client + 5-min in-memory name cache. Exports: `getLangfuse()`, `resolveOrgName()`, `resolveCustomerInfo()`, `resolveCampaignName()`, `trackCallDispatch()`, `trackGeneration()`, `trackCallCompletion()`, `flushLangfuse()`.
`client_backend/src/services/openai.service.ts`	Extracts token usage from OpenAI response (`input_tokens`, `output_tokens`, `total_tokens`) and passes to `trackGeneration()` for `hard_words_extraction`.
`client_backend/src/services/calling.service.ts`	Resolves org/customer/campaign names before dispatch. Passes names in `DispatchMetadata`. Calls `trackCallDispatch()` after each call dispatch.
`client_backend/src/services/livekit.service.ts`	Added `org_name`, `customer_name`, `customer_email`, `campaign_name` to `DispatchMetadata` interface.
`client_backend/src/controllers/webhook.controller.ts`	Calls `trackCallCompletion()` on `participant_left` with billing duration, call status, disconnect reason.
`client_backend/src/controllers/agent.controller.ts`	Test call dispatch resolves org/customer names via `resolveOrgName()` and `resolveCustomerInfo()` before dispatch.

What Data Goes to Langfuse

6 things are sent to Langfuse:

#	Trace Name	Source	What it contains	Where it shows in Langfuse
1	`call_dispatch`	client_backend `calling.service.ts`	Room name, contact name, phone, campaign, agent, org/customer names	Tracing tab → metadata. Tags: `dispatch`, `org:`, `customer:`, `campaign:`
2	`call_completion`	client_backend `webhook.controller.ts`	Billing duration (seconds), call status, disconnect reason, org/customer/campaign names	Tracing tab → metadata. Tags: `call_completion`, `org:`, `customer:`, `campaign:`
3	`hard_words_extraction`	client_backend `openai.service.ts`	OpenAI input/output tokens, model, prompt, extracted words	Tracing tab → has a `generation` observation with token counts
4	LiveKit auto-spans	Agent `telemetry.py` (OTel)	Individual STT/LLM/TTS spans per turn latency, model, tokens (LLM only)	Tracing tab → auto-created traces with observations
5	`call_usage_summary`	Agent `post_call.py`	Aggregated per-call: LLM tokens, TTS characters, STT duration, all org/customer/campaign metadata	Tracing tab → has `voice_call_llm` generation with native token counts
6	`conversation_analysis`	Analysis `analysis_orchestrator.py`	OpenAI tokens/cost from analysis, org/customer/campaign metadata	Tracing tab → has `structured_json_call` generation with auto-captured tokens

Every trace carries these attributes:

Attribute	Langfuse field	Value
Organization UID	`User Id`	org UID (e.g. `org_zWFf1nZLL1N8CFut`)
Organization Name	`metadata.orgName`	Human-readable name
Customer UID	Tag `customer:{uid}` + `metadata.customer_id`	Customer UID
Customer Name	`metadata.customerName`	Human-readable name
Customer Email	`metadata.customerEmail`	Email address
Campaign UID	Tag `campaign:{uid}` + `metadata.campaign_id`	Campaign UID
Campaign Name	`metadata.campaignName`	Human-readable name
Call Session	`Session Id`	LiveKit room name (e.g. `room-5c1d78bd`)

What Contributes to Tokens & Cost

Only generation type observations with token counts feed into Langfuse's Total Tokens and Total Cost metrics.

Generation	Trace	Tokens?	Cost?
`voice_call_llm`	`call_usage_summary`	Yes input + output tokens	Yes (after model pricing configured)
`voice_call_tts`	`call_usage_summary`	No uses CHARACTERS unit	Only if character-based pricing configured
`voice_call_stt`	`call_usage_summary`	No uses MILLISECONDS unit	Only if time-based pricing configured
`structured_json_call`	`conversation_analysis`	Yes auto-captured from OpenAI	Yes OpenAI pricing is built-in
`hard_words_extraction`	`hard_words_extraction`	Yes input + output tokens	Yes OpenAI pricing is built-in
LiveKit LLM spans	Auto-created	Yes `gen_ai.usage.*` OTel attributes	Yes (after model pricing configured)

For your core requirement (total input tokens, output tokens, cost per call/customer/org):

LLM tokens come from voice_call_llm + structured_json_call + hard_words_extraction
Filter by User Id for per-org, by customer: tag for per-customer, by Session Id for per-call

The other traces (call_dispatch, call_completion) are metadata-only they show call lifecycle info but contribute zero to token/cost numbers.

Dashboard Setup (Step by Step)

Langfuse version: v3.160+ (Hobby plan or above)

Step 1: Create the Dashboard

Go to https://cloud.langfuse.com → select your project
Left sidebar → Dashboards
Click "New Dashboard" → name it Oktivo Operations

Step 2: Add Widgets

Click "+ Add Chart" for each widget. Here are the exact settings:

Widget 1: Cost by Organization

Setting	Value
View	`Traces`
Metric	`Total Cost`
Breakdown Dimension	`User Id`
Chart Type	`Horizontal Bar Chart`
Name	`Cost by Organization`

Each bar = one org's total LLM cost. Shows $0.00 until custom model pricing is configured.

Widget 2: Cost by Customer

Setting	Value
View	`Traces`
Metric	`Total Cost`
Breakdown Dimension	`Tags`
Chart Type	`Horizontal Bar Chart`
Name	`Cost by Customer`

Shows cost grouped by all tags. Look for customer: prefixed entries for per-customer cost.

Widget 3: Tokens by Organization

Setting	Value
View	`Traces`
Metric	`Total Tokens`
Breakdown Dimension	`User Id`
Chart Type	`Horizontal Bar Chart`
Name	`Tokens by Organization`

Widget 4: Cost by Model

Setting	Value
View	`Observations`
Metric	`Total Cost`
Breakdown Dimension	`Model`
Chart Type	`Pie Chart`
Name	`Cost by Model`

Use Observations view (not Traces) because model info lives on individual observations.

Widget 5: Token Usage Over Time

Setting	Value
View	`Traces`
Metric	`Total Tokens`
Breakdown Dimension	`None`
Chart Type	`Line Chart` (Time Series)
Name	`Token Usage Trend`

Widget 6: Total Calls by Organization

Setting	Value
View	`Traces`
Metric	`Count`
Breakdown Dimension	`User Id`
Chart Type	`Horizontal Bar Chart`
Name	`Calls by Organization`

Widget 7: Daily Call Volume

Setting	Value
View	`Traces`
Metric	`Unique Session Ids`
Breakdown Dimension	`None`
Chart Type	`Line Chart` (Time Series)
Name	`Daily Call Volume`

Each unique session = one call (room name).

Widget 8: Latency by Model

Setting	Value
View	`Observations`
Metric	`Latency`
Breakdown Dimension	`Model`
Chart Type	`Horizontal Bar Chart`
Name	`Latency by Model`

Shows which provider (Deepgram, Gemini, Cartesia) is slowest.

Widget 9: Trace Types Breakdown

Setting	Value
View	`Traces`
Metric	`Count`
Breakdown Dimension	`Name`
Chart Type	`Pie Chart`
Name	`Trace Types`

Shows distribution: call_dispatch, call_completion, conversation_analysis, call_usage_summary, hard_words_extraction.

Widget 10: Active Organizations

Setting	Value
View	`Traces`
Metric	`Unique User Ids`
Breakdown Dimension	`None`
Chart Type	`Big Number` (Total Value)
Name	`Active Organizations`

Widget 11: Activity by Tag

Setting	Value
View	`Traces`
Metric	`Count`
Breakdown Dimension	`Tags`
Chart Type	`Horizontal Bar Chart`
Name	`Activity by Tag`

Shows counts for all tags: org:, customer:, campaign:, dispatch, analysis, call_completion, usage_summary.

Drill-Down: Org → Customer → Campaign → Call

See all traces for one organization

Left sidebar → Tracing
Add filter: User ID = org UID
All traces for that org appear with cost/token columns

See one customer within an org

Same Tracing tab with org filter
Add filter: Tags → contains customer:{customerUID}
Or: filter by Metadata → key: customerName, value: the name

See one campaign

Add filter: Tags → contains campaign:{campaignUID}
Or: filter by Metadata → key: campaignName, value: the name

See one call (full lifecycle)

Left sidebar → Sessions
Click on a session (room name like room-5c1d78bd)
Shows all traces for that call grouped together:
- call_dispatch call initiated
- LiveKit STT/LLM/TTS spans per-turn details
- call_usage_summary aggregated LLM tokens
- conversation_analysis analysis OpenAI tokens
- call_completion billing duration, status

Agent + Analysis traces share the same session_id (room name), so they appear together.

Custom Model Pricing (Fixes $0.00)

Langfuse auto-prices OpenAI models. For all other providers, you must add pricing manually.

Left sidebar → Settings (bottom)
Go to "Models" tab
Click "Add Model" for each:

Model name (must match exactly)	Provider	Pricing source
`gemini-2.5-flash`	Google	https://ai.google.dev/pricing
`nova-3`	Deepgram (STT)	https://deepgram.com/pricing
`sonic-turbo`	Cartesia (TTS)	https://cartesia.ai/pricing
`saaras:v3`	Sarvam (STT)	https://sarvam.ai/pricing
ElevenLabs model	ElevenLabs (TTS)	https://elevenlabs.io/pricing
`Falcon`	Murf (TTS)	https://murf.ai/pricing

Tip: Before adding pricing, go to Tracing → click any observation → check the Usage section to see what unit Langfuse received (tokens, characters, milliseconds). Match your pricing unit to that.

Once configured, all cost widgets and columns show real dollar amounts.

Filter Cheat Sheet

I want to see...	Filter
Everything for org X	`User ID` = org UID
Customer Y within org X	`User ID` = org UID + `Tags` contains `customer:{uid}`
Customer Y by name	`Metadata` → key: `customerName`, value: name
Campaign Z	`Tags` contains `campaign:{campaignUID}`
Campaign Z by name	`Metadata` → key: `campaignName`, value: name
Single call	`Session ID` = room name
Only analysis traces	`Tags` contains `analysis`
Only dispatches	`Tags` contains `dispatch`
Only completions	`Tags` contains `call_completion`
Only usage summaries	`Tags` contains `usage_summary`
Full call lifecycle	`Session ID` = room name (Agent + Analysis traces share this)

Adding Observability to New Code

Python New LLM call

python

from langfuse import observe

class MyService:
    @observe(as_type="generation")
    async def my_llm_call(self, prompt: str):
        response = await openai_client.responses.create(
            model="gpt-4o-mini",
            input=[{"role": "user", "content": prompt}],
        )
        return response.output[0].content[0].text

Python Tag with org metadata

python

from langfuse import observe, propagate_attributes

@observe(name="my_operation")
async def my_function(self, org_id: str):
    with propagate_attributes(
        user_id=org_id,
        tags=[f"org:{org_id}"],
        metadata={"organization_id": org_id},
    ):
        # child spans inherit these attributes
        pass

Node.js New tracked event

typescript

import { getLangfuse } from "./langfuse.service.js";

const lf = getLangfuse();
if (lf) {
  const trace = lf.trace({
    name: "my_operation",
    userId: orgUID,
    tags: [`org:${orgUID}`],
  });
  trace.generation({
    name: "my_llm_call",
    model: "gpt-4o-mini",
    input: prompt,
    output: result,
    usage: { input: inputTokens, output: outputTokens },
  });
  await lf.flushAsync();
}

Troubleshooting

Problem	Fix
No traces appearing	Verify `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` are set correctly
All costs $0.00	Configure custom model pricing: Settings → Models (see section above)
Agent traces missing org metadata	Check `metadata_resolver.py` maps `orgUID` → `organization_id`
Analysis traces have no token counts	Ensure `@observe(as_type="generation")` is on the OpenAI call
Analysis traces not linked to call	Ensure `room_name` is passed from Agent to Analysis
Traces in wrong project	Wrong API keys each env needs its own project keys
Data not flushing	Check `flush_langfuse()` / `lf.flushAsync()` in shutdown handlers

Langfuse Observability Oktivo ​

Table of Contents ​

Environment Setup ​

Architecture Overview ​

Files Changed for Langfuse ​

Agent Service (Python) ​

Analysis Service (Python) ​

client_backend Service (Node.js) ​

What Data Goes to Langfuse ​

6 things are sent to Langfuse: ​

Every trace carries these attributes: ​

What Contributes to Tokens & Cost ​

Dashboard Setup (Step by Step) ​

Step 1: Create the Dashboard ​

Step 2: Add Widgets ​

Drill-Down: Org → Customer → Campaign → Call ​

See all traces for one organization ​

See one customer within an org ​

See one campaign ​

See one call (full lifecycle) ​

Custom Model Pricing (Fixes $0.00) ​

Filter Cheat Sheet ​

Adding Observability to New Code ​

Python New LLM call ​

Python Tag with org metadata ​

Node.js New tracked event ​

Troubleshooting ​