Evaluation & Quality

Score agent outputs, monitor quality, and route reviews.

Judge whether agents produce good output. Define evaluators, monitor production executions in Online Eval, and route low-confidence cases to Human Review.

Evaluators, the named scoring methods available to your agents.

Configure quality checks

Create evaluators that score executions with a judge model and confidence threshold.

Review uncertain outcomes

Watch live results, then resolve tasks routed to a person.

At a glance

Area	Use it to	Start here
Evaluators	Define scoring methods and routing thresholds	Evaluators
Online Eval	Monitor scored production executions	Online Eval
Human Review	Resolve low-confidence review tasks	Human Review

An evaluator scores each execution and reports its confidence. High-confidence results are accepted automatically. Low-confidence results route to Human Review for a person to decide.

Evaluators

An evaluator is a named scoring method backed by a judge model. Open Evaluation & Quality → Evaluators to see the ones your account has and to create more.

At the top, the page summarizes your setup: how many evaluators exist and how many are active, how many judge models are available across Anthropic, OpenAI, and Google, and how many agents have online eval running. Two tabs sit below — Evaluators, the list, and Agent enablement, where you turn evaluators on for each agent.

Each evaluator is listed with:

Field	What it shows
Evaluator	The evaluator's name, its evaluation method, and how many agents use it.
Framework	How it scores — Single judge (one model scores each output).
Judge model(s)	The model doing the scoring (for example, gpt-5.1).
Conf. → HITL	The confidence threshold below which results route to human review (for example, < 67%).
Status	Whether the evaluator is Active.
Actions	A per-row menu to edit or remove the evaluator.

Create an evaluator

1

Open the creation dialog

Click + New evaluator.

2

Configure the evaluator

Fill in the following fields:

• Evaluator name — required. A label for this evaluator (for example, Answer relevancy).

• Framework — Single judge scores each output with one model. Jury vote (several judges, weighted) is marked Coming soon and can't be selected yet.

• Evaluation method — the platform method to score against: Answer Relevancy, Safety, or Task Completion.

Configuring a new evaluator.

Choosing the evaluation method to score against.

• Judge model — required. The model that does the judging.

• Confidence threshold — a slider, defaulting to 50%. The judge returns a rating together with a self-reported confidence from 0–100%. Any execution whose confidence falls below this threshold is flagged for human review regardless of its rating.

• Route low-confidence results to human review — a per-evaluator preference for sending below-threshold results to Human Review.

3

Save the evaluator

Click Create evaluator to save. The button stays disabled until the required fields — name, evaluation method, and judge model — are set.

Online Eval

Online Eval scores live production traffic. Open Evaluation & Quality → Online Eval to see every execution an evaluator has scored and where it went.

Online Eval, scoring live production executions.

You can narrow the view by agent, by minimum score, by verdict, or by routing, and search by execution, agent, or evaluator. Across the selected window you see how many executions were scored, how many Passed at or above the pass threshold, how many were Routed to Human Review below judge confidence, and the average score, with an Open board shortcut into Human Review.

Each row is one scored execution:

Column	What it shows
Execution	The execution's ID.
Agent	Which agent produced the output.
Evaluator	Which evaluator scored it (for example, PII Mask, Test valuator).
Qual.	The quality result — a pass or a percentage score.
Conf.	The judge's confidence, shown against the evaluator's threshold (for example, 0.96 ≥ 1.00).
Verdict	Passed or Failed.
Routing	Auto-accepted or Human Review.

Routing depends on confidence, not the verdict

An execution is routed to Human Review when the judge's confidence falls below the evaluator's threshold — even if its verdict is Passed.

Conversely, a Failed execution whose confidence is above threshold is Auto-accepted. Routing asks "is the judge sure?", not "did it pass?".

Human Review

Human Review is the workspace where a person records a verdict on the executions that scored below an evaluator's confidence. Open Evaluation & Quality → Human Review. Tasks move from Review Pending (waiting on a reviewer's verdict) to Reviewed (verdict recorded).

Human Review, the queue of tasks awaiting a reviewer's verdict.

Switch between My tasks, All tasks, and Unassigned, and narrow by team, type, status, or priority. Your queue is summarized as Assigned to me, Reviews in view, Reviewed, and Unassigned, with a View unassigned shortcut. Reviews are assigned through teams, so a reviewer who belongs to no team has nothing assigned to them.

Each task row shows the Execution, the Eval type and Evaluator that flagged it, the Team and Assignee responsible, its Priority, when it was Queued, its Status, and the recorded Verdict. Open a task to record your verdict and move it to Reviewed.

Notes

• Single judge is the only framework available today; Jury vote is Coming soon.

• Confidence threshold is set per evaluator, so different evaluators can route to human review at different confidence levels (< 67%, < 100%, and so on).

• Human Review assignments flow through teams — set up in Account Settings — not to individuals directly.

Related

Control CenterObservabilityControls & AuditAccount Settings