Online Eval

Monitor evaluator scores for live production executions.

Online Eval scores live production traffic. Open Evaluation & Quality → Online Eval to view scored executions and their routing.

Online Eval scores live production executions.

Filter by agent, minimum score, verdict, or routing. Search by execution, agent, or evaluator.

For the selected time range, the summary shows:

• Scored executions

• Executions that passed at or above the pass threshold

• Executions routed to Human Review below judge confidence

• Average score

Select Open board to open Human Review.

Scored execution fields

Column	What it shows
Execution	The execution ID.
Agent	The agent that produced the output.
Evaluator	The evaluator that scored it, such as PII Mask.
Qual.	The quality result: pass or a percentage score.
Conf.	The judge confidence and evaluator threshold, such as 0.96 ≥ 1.00.
Verdict	Passed or Failed.
Routing	Auto-accepted or Human Review.

Routing depends on confidence, not the verdict

An execution routes to Human Review when confidence falls below the evaluator threshold. This can happen even when the verdict is Passed.

A Failed execution is auto-accepted when confidence meets the threshold. Routing reflects judge confidence, not the verdict.