Debug & tracing

A detailed debug view — every prompt, tool call, MCP frame, model param, token count and latency of a turn. This is telemetry, not the source of truth: append-only, write-heavy, high-volume, read mostly for debugging and eval, with shorter retention than the chats.

Rule: traces do not go in the transactional Postgres. Raw prompts/traces would bloat the chat DB and mix telemetry with truth. They get their own system.

What exists today is not this

Langfuse is not deployed, and nothing here is instrumented for it. What a turn's prompt can actually be read from today is the prompt log — a table, off by default, switched on per agent by that agent's owner, kept seven days, and readable only by a platform administrator in the panel. The short, always-on record of what an agent did is a different thing again. Both are described in What the agent did. The rest of this page is the plan.

Recommended: self-hosted Langfuse

Langfuse is built for exactly this — and gives the debug UI out of the box:

• Traces & sessions — the full tree of a turn: prompts, LLM calls, tool calls / MCP traffic, intermediate steps, nested sub-agents.
• Prompts, tokens, cost, latency per call; search & filter; compare runs.
• Eval — datasets, scores, regression checks later.
• Self-hosted in the cluster (Postgres + ClickHouse for the high-volume trace store).

The brain (api) is instrumented to emit a trace per turn; the worker's tools/call and results attach as spans. Nothing about this lives in the chat table.

Alternative (roll-your-own): write trace events to ClickHouse (or JSONL in object storage) — cheap columnar storage for append-only logs — but then you build the UI yourself. Langfuse wins for the debug experience.

Retention & tiers

• Hot in ClickHouse for recent debugging (days–weeks).
• Cold rolled to object storage (or dropped) on a retention policy — traces are disposable in a way chats are not.

What surfaces where

• This trace plane is internal (debug/eval), the same control-plane data that never reaches the user chat — see Worker → Use for the chat-vs-control-plane split.
• A turn carries a trace id; the chat Message can store that id so you can jump from a message straight to its full trace.

See also

• Storage — the chat of record (kept free of trace blobs).
• Worker → Use — the MCP tool traffic that becomes spans.
• Implementation — wiring the instrumentation.