Online-Eval
Score every live conversation automatically, in the background, without affecting end users.
Online Evals is the production-side evaluation system on the Uptiq Platform. Once an agent is deployed, it runs by itself on real conversations, without anyone asking it to. The grading happens off to the side — not in the path of the actual response. If the grading itself fails for any reason — a timeout, an error — that failure gets logged and retried, but it never touches the live conversation. The user never sees it, never feels it.
Background scoring
Runs automatically on completed conversations, off the critical path
Configurable scorers
Define which quality dimensions are evaluated, with custom prompts
Performance dashboard
Current scores, trends over time, and drill-down to specific conversations
Threshold alerting
Get alerted when quality drops past a threshold you've set
Feedback loop
Production problems feed back into the next round of building, done manually
Failure isolation
Grading failures are logged and retried — never touch the live conversation
Model-provider agnostic
Works regardless of which LLM powers the agent or the grader
Online Evals is the runtime counterpart to Evals. Use Evals to validate quality before deployment, Online Evals to score live traffic after deployment, and Reinforcement Learnings to turn reviewed feedback into approved learning cases.
Online Evals is accessed from the Cognition & Control section in the Agent Builder's "Manage Agent" modal sidebar. The interface has two tabs:
Scorers — Configure which quality dimensions are evaluated
Runs — View evaluation run history and results
How It Works
Once the agent is live, Online Evals takes over automatically:
End users invoke the agent in real conversations
Online eval runs in the background — non-intrusive
Performance results stream to the Maintainer's dashboard
Maintainer tracks agent performance over time
Performance drift or regression?
No → Keep watching
Yes → Iterate and refine the agent — feeds back to the builder
If something's drifted, that loops back to whoever built the agent, so production problems actually feed back into the next round of building — not just into a report nobody reads.
Dashboard
What comes out of Online Evals lands on a dashboard for whoever's maintaining the agent:
Current performance — How the agent is scoring right now
Threshold suggestions — If quality drops past a threshold score, suggestions pop up
Drill-down — Click into a metric and see exactly which real conversations are dragging the score down
Guardrails and Governance
Who can do what is enforced at the platform level — it's a permission boundary, not a process people are trusted to follow. Builders get the offline/testing side, Maintainers get the live monitoring side.
Whatever real conversation data feeds the online evals still respects tenant isolation and PII rules — it's not an open pipe into live data.
Online Evals works the same regardless of which LLM provider is behind either the agent or the grader.
Related sections
Monitoring — Track production health and investigate runtime issues.
Evals — Validate changes against test datasets before rollout.
Reinforcement Learnings — Review approved corrections from real conversations.
Last updated