Online Eval
Monitor evaluator scores and runs for live production executions.
Online Eval scores live production traffic. Open Control Center → Evals → Online Eval. The page has two tabs:
Scores lists every score an evaluator produced, with its verdict and routing.
Runs lists every evaluator run, including the ones that never produced a score, so you can see why a score is missing.
Scores
Filter by project, agent, minimum score (Score ≥), verdict, or routing. Search by execution, agent, or evaluator.
The summary counts what the filters leave in view:
Evaluations in view: every score that matches the filters.
Passed: scores at or above the pass threshold, with their share of the total.
Routed to Human Review: scores below judge confidence that were sent to a reviewer, with their share.
Avg. score: the average score, with the average confidence beneath it.
Below the summary, the same counts are drawn as a flow from Scored live to Passed and Routed to Human Review. Select Open board to open Human Review.
Scored execution fields
Execution
The execution ID.
Agent
The agent that produced the output.
Evaluator
The evaluator that scored it, such as Jury Safety.
Score
The result: pass or fail for a pass/fail method, or a percentage.
Conf.
The judge's confidence, with the evaluator's confidence threshold beneath it. 87% over ≥ 95% means a confidence of 87% against a threshold of 95%.
Verdict
Passed or Failed.
Routing
Auto-accepted or Human Review.
Scored
When the score was recorded, such as 7 Hrs ago.
Routing depends on confidence, not the verdict
An execution routes to Human Review when its confidence falls below the evaluator's threshold and the evaluator has Route low-confidence results to human review turned on (see Evaluators). This can happen even when the verdict is Passed.
A Failed execution is auto-accepted when confidence meets the threshold. Routing reflects judge confidence, not the verdict.
Runs
When an agent's live conversation is sampled, Qore creates one run for each evaluator assigned to that agent. A run that scores shows up on Scores. A run that fails or is skipped has no score, so Runs is the only place it appears.
Filter by project, agent, or status, and search by execution. Each row shows the Execution, Agent, Evaluator, Status, and when the run was Created.
Pending — the run is waiting to start.
Running — the judge model is scoring the conversation.
Completed — the run produced a score, which appears on Scores.
Failed — the run ended without a score. Hover over the status to see why:
Judge model unavailable: <model>— the evaluator uses a judge model that isn't available.Evaluation method not found— the evaluator's method no longer exists.Evaluation timed out— the run stayed Pending or Running for more than two hours.Evaluation failed— scoring hit an error.
Skipped — the run was never scored, either because its evaluator was deleted before scoring started or because the conversation had nothing to score.
Judge reasoning and evidence
Expand a score row to see the details behind it. Judge reasoning is the judge's explanation of the score. If the judge gave none, it reads The judge did not return a justification for this score.
Below it, Evidence lists up to three points behind the score, ordered from the one that moved it most to the one that moved it least. Each point cites something specific the agent did or failed to do, and says whether it raised or lowered the score. A judge can return fewer points, or none, and then no Evidence list is shown. For a Jury evaluator, each judge shows its own reasoning and evidence.
Open the linked conversation
Open linked conversation, labelled with the run's execution ID, opens Runs & Traces filtered to that one conversation, so you can read exactly what the agent did. The filtered view survives a reload, so you can share its link with a colleague.
Export results
Online Eval has no export button of its own. To download evaluation results, open Control Center → Observe → Export and choose the Eval Results dataset: scores, verdicts, reasoning, evidence, and human overrides for evaluated agent runs.
The Export page sets the project scope, date range, format, and delivery, and can run an export once or on a schedule. Large exports run in the background. See Export.
What the judge scores
Each evaluator scores a transcript of the conversation. The transcript includes:
every message, with its formatting kept inside the turn it belongs to
the documents attached by the user and produced by the agent, on each turn
each tool call's arguments, whether it succeeded, and what it returned
rich replies, such as widgets, rendered as text
Long tool results are shortened, and the transcript says where content was cut. A conversation too long for the judge model is trimmed to whole turns, and the transcript notes how many turns were left out.
This matters when you read a score. A judge sees what the tools returned, so an answer built on a failed tool call can score differently from the same answer built on a successful one.
Maintained by Abhishek Paul
AI-assisted, human-approved
Last updated
Was this helpful?