Evaluate agent quality

Every Evals task in Control Center: create evaluators, score agents' live traffic, read scores and runs, and review low-confidence results.

This guide is organized by task. Find what you want to do in the list below, then follow the steps. Each task links to the reference page it comes from.

An evaluator is a named scoring method backed by a judge model, defined at the account level. You enable evaluators per agent to score its live traffic, and low-confidence results can go to people for review.

Task	Go to
See the account's evaluators and judge models	Evaluators
Create an evaluator	Create
Choose between one judge and a jury	Framework
Edit or remove an evaluator	Edit
Choose which evaluators score an agent	Assign
Start scoring an agent's live traffic	Turn on
Score only part of an agent's traffic	Sampling
Read the scores	Scores
See why a result went to Human Review	Routing
See the judge's reasoning and evidence	Reasoning
Find out why a score is missing	Runs
Read the conversation behind a score	Linked conversation
Review a low-confidence result	Human Review
Export evaluation results	Export

---

Evaluators

See the evaluators

Open Evals › Evaluators. The summary shows how many Evaluators exist and are active, how many Judge models are available and from which providers, and the Online eval agents scoring live traffic out of all agents, such as 9 / 71.

Each evaluator row shows its name and Active status, its method and how many agents use it, its Framework, its Judge model(s), its Confidence Threshold, and an Actions menu.

Evaluators.

Create an evaluator

1

Open the dialog

Select + New evaluator. You need the Build Evals capability; without it the button is disabled.

2

Name it and choose the framework

• Evaluator name — required, such as Answer relevancy.
• Framework — Single judge or Jury.

3

Choose the method

Evaluation method: Answer Relevancy, Faithfulness, Safety, or Task Completion.

4

Choose the judge models

Judge model for a single judge, or Jury models (2 to 5, from any providers) for a jury.

5

Set the threshold and routing

• Confidence threshold — default 50%. Results below it are flagged, regardless of rating.
• Route low-confidence results to human review — off by default. While it's off, low-confidence results are still scored but auto-accepted.

6

Save

Select Create, or Create & assign agents to go straight to choosing the agents it scores. Both are enabled once you set a name, a method, and the judge model or jury models.

New evaluator.

Faithfulness rates how much of what the agent said is supported by its context: retrieved context, tool results, and what the user said. The score is supported claims divided by all factual claims. Reference: Evaluators › Faithfulness.

Choose between a single judge and a jury

Framework	Judge models	Scoring
Single judge	One model	That model scores each output directly.
Jury	2 to 5 models	Each judge scores independently with its own confidence; Qore combines them into one weighted verdict.

Each evaluator's confidence threshold is independent of every other evaluator's. Reference: Evaluators › Choose a framework.

Edit or remove an evaluator

Use the evaluator's Actions menu.

---

Score live traffic

Assign evaluators to an agent

1

Open Agent enablement

On Evaluators, select the Agent enablement tab. Filter by project, or search for an agent. An agent with the same name in two projects has two rows.

2

Choose the evaluators

Select Edit in the agent's Assigned evaluators cell. Assign evaluators lists every evaluator in the account. Tick the ones that should score this agent, then select Save.

Assign evaluators.

Assigning evaluators doesn't start scoring on its own.

Turn on Online Eval for an agent

On Agent enablement, turn on the agent's Online eval toggle. While it's on, every enabled evaluator assigned to the agent scores the sampled share of its conversations.

An agent can show Unsupported instead of a toggle: its type doesn't support Online Eval yet. You can still pre-assign evaluators with Edit.

Agent enablement.

Score only part of the traffic

On Agent enablement, set the agent's Sampling: the chance that each live conversation is evaluated. The default is 100%; lower it on high-traffic agents to keep judge costs down. Online Eval scoring is billed as OnlineEval on Credit Usage › Features.

Reference: Evaluators › Agent enablement.

---

Read results

Read the scores

Open Evals › Online Eval, on the Scores tab. Filter by project, agent, minimum score (Score ≥), verdict, or routing, and search by execution, agent, or evaluator.

The summary counts what's in view: Evaluations in view, Passed, Routed to Human Review, and Avg. score with average confidence. Each row shows the Execution, Agent, Evaluator, Score (pass/fail or a percentage), Conf. (confidence over the threshold, such as 87% over ≥ 95%), Verdict, Routing, and Scored time.

Online Eval, Scores tab.

See why a result went to Human Review

An execution routes to Human Review when its confidence is below the evaluator's threshold and the evaluator has Route low-confidence results to human review on. This can happen even when the verdict is Passed. A Failed execution is auto-accepted when confidence meets the threshold.

See the judge reasoning and evidence

Expand a score row. Judge reasoning explains the score. Evidence lists up to three points, from the one that moved the score most, each citing something the agent did or failed to do. For a Jury evaluator, each judge shows its own.

Find out why a score is missing

Open the Runs tab. Each sampled conversation creates one run per assigned evaluator. Filter by project, agent, or status, and search by execution. Statuses are Pending, Running, Completed, Failed, and Skipped. For Failed, hover over the status for the reason:

• Judge model unavailable: <model>
• Evaluation method not found
• Evaluation timed out — the run stayed Pending or Running for over two hours.
• Evaluation failed

Skipped means the evaluator was deleted before scoring, or the conversation had nothing to score.

Read the conversation behind a score

Select Open linked conversation, labelled with the run's execution ID. It opens Runs & Traces filtered to that conversation; the filtered view survives a reload, so you can share its link.

Reference: Online Eval.

---

Review

Review a low-confidence result

1

Open Human Review

Open Evals › Human Review, or select Open board on Online Eval.

2

Choose a queue

Switch between My tasks, All tasks, and Unassigned, and filter by team, type, status, or priority. View unassigned opens unassigned tasks.

3

Record a verdict

Open a task. Its evaluation summary shows the judge's reasoning and evidence. Record a verdict; the task moves from Review Pending to Reviewed.

Human Review.

Human Review assigns tasks through teams, not to individuals. A reviewer without a team has no assigned tasks. Set up teams in Manage users and access › Create a team.

Reference: Human Review.

Export evaluation results

Online Eval has no export button of its own. Open Observe › Export and choose the Eval Results dataset: scores, verdicts, reasoning, evidence, and human overrides. See Monitor activity › Export.

---

Related pages

• Evals, Evaluators, Online Eval, and Human Review
• Test an agent before release: Create an agent › Build a test dataset

---

Maintained by Abhishek Paul AI-assisted, human-approved