Evaluators
Create scoring methods, set a pass mark, and decide which results a person reviews.
Evaluators are named, account-level scoring methods backed by a judge model. They help you apply consistent quality standards across agents, set a pass mark, and route uncertain or failed results for review.
Open Control Center → Evals → Evaluators to view, edit, or create evaluators. An evaluator isn't created per agent or per project, so the same one can serve every agent that enables it. Attach an evaluator to individual agents in Agent List. For task-focused instructions, see Create an evaluator in Task Guides.
Permissions
+ New evaluator
The Build Evals capability. Without it, the button isn't available.
Evaluators table
The table lists your evaluators, with Search and a + New evaluator button. Type part of an evaluator's name in Search and press Enter to narrow the table.
Evaluator
The evaluator name.
Type
What the evaluator returns, or who scores: Checklist, Single score, Pass or fail, or Agent.
Method
The evaluation method, such as Answer Relevancy. A dash for an evaluator that an agent scores.
Scored by
The framework and what scores: Single judge with the name of its model, Jury with the names of its models, or Agent with the agent's name.
Pass mark
The lowest score that passes, shown with ≥ and a percentage. Methods that return only pass or fail show Pass / fail.
Review rule
Which results a person also checks, shown as Confidence < and a percentage, or Off for an evaluator whose routing to the Review Queue is turned off.
Agents
How many agents the evaluator scores.
Status
Active for an evaluator that is in use.
Actions
A menu with Edit and Assign agents.
An evaluator only scores the agents assigned to it. Assigning alone doesn't start scoring: the agent's Online eval switch must be on too. See Agent List.
Evaluation methods
Choose a method based on the quality signal you need. The method determines what the judge evaluates and whether it returns a score or a pass-or-fail result.
Checklist
Whether the agent satisfied each checklist item you write. Items can be weighted and marked as critical.
A result per check, combined into a weighted score.
Answer Relevancy
How much of the agent's answer addresses what the user asked.
Single score
Faithfulness
How much of the agent's answer is supported by the context it received.
Single score
Safety
Whether the agent avoided harmful content and kept personal data out of its responses.
Pass or fail
Task Completion
Whether the agent accomplished what the user was trying to do.
Single score
Frameworks
Single judge
One model
That model scores each output directly.
Fast, low-cost scoring where one capable model's judgment is enough.
Jury
Several models
Each judge model scores independently and reports its own confidence as a percentage from 0 to 100. Qore combines the votes into one weighted verdict.
Higher-stakes evaluation methods, where no single model's judgment should decide the outcome alone.
Agent
One of your agents
The agent scores each output and replies with a score, a confidence, and its reasoning. See Agent judge.
Scoring that follows an agent you have built.


Evaluator configuration
The New Evaluator dialog has five steps, shown as tabs: Method, Basics, a step named for the method, Gates and routing, and Agents. The method step is Checklist for a checklist and Method details for the other methods. Back and Next move between them.
Method
Method reads "Start with how it should judge. Everything else comes with sensible defaults." Each method is a card with a one-line description. Checklist is first, marked Recommended, and selected by default.

Basics
Basics identifies the evaluator and its judge. Next isn't available until you enter an Evaluator name.
Evaluator name
Required.
Framework
Single judge uses one model. Jury uses several judge models and takes a weighted vote.
Judge model
Required for Single judge.
Jury models
Required for Jury. Select 2 to 5 models. Models can come from different providers. The picker is a searchable list with provider filters, and it starts with one model selected. It displays "Pick 2–5 models," and Next isn't available until you select two to five.
Method details
Method details describes what a non-checklist method asks and returns. The platform manages these values, so you change nothing here.
Judge question
The question used to evaluate the response. For Task Completion: "Did the agent complete the task it was given?"
Output
A Single score from 0 to 100, with a written reason.
Confidence
One figure for the whole verdict.
Prompt
Managed by the platform.
Switch to Checklist gives each check its own result, with weights and Critical checks.
Gates and routing
Gates and routing reads "When an output passes, and when a person reviews it." A person always checks every failed output, every Critical miss, and any output the judge is unsure about.
Pass mark
Pass at or above a percentage. Below this score, an output fails. A pass-or-fail method has no score to set.
Human review
Confidence below a percentage, or Juror agreement below for a jury. Any verdict below this is reviewed. In Edit, a jury's control is labeled Confidence below.
Review never changes the Pass or Fail result.
Confidence is how sure the judge is of its verdict. For a checklist, it is the lowest confidence across the checks. For a jury, it is how many jurors agree with the majority.
Agents
Agents lists each agent with its project and evaluator count. Select all and Clear change the selection. You can also assign agents later. The button reads Create evaluator. After you select agents, it changes to, for example, Create and assign to 1 agent, and saving assigns them.
Review routing
An evaluator you create in the dialog always routes to Review Queue. Evaluators created earlier can have routing turned off. Their Review rule reads Off, and none of their results go to the Review Queue.
In Edit, Review when the judge is unsure is a switch. Turn it off to stop routing for the evaluator. The Confidence below control disappears, and the Review rule reads Off. Pass threshold and Confidence below each have a slider and a number box. Under them, the dialog notes that failed executions always go to human review, and, for a checklist, critical check misses too.
Checklist
Checklist scores whether the agent satisfied each checklist item you write, with weighted and critical items. An evaluator can have up to 10 checks, and at least one.
Check
A name for the check.
What the judge looks for
The criterion, written as a statement the judge can verify.
Weight
How much the check counts. Weights are scaled to 100% across the checks that apply, so a weight of 30 next to one of 20 gives shares of 60% and 40%. The default is 10.
Critical
A switch. Missing a critical check fails the output outright, whatever the score, and sends it to a person.
The judge rules on every check. The score is the total weight of the checks that passed divided by the total weight of all checks. The output passes only when the score reaches the pass mark and no critical check was missed. A missed critical check fails the evaluation whatever the score, and it always goes to Review Queue.
Checks in the dialog
New Evaluator, Checklist step
Counts your checks and your Critical checks, and draws a bar of their shares. The table lists each check with its Share. A check's checkbox turns it on or off. When you clear it, the check is left out of scoring, its Share reads a dash, and the other checks take its share. A row's Actions menu has Edit and Remove. Add check takes a Weight from 1 to 50.
Edit
Shows how many checks are turned on out of the total, with each check's Weight and a Critical badge, and Edit and Remove icons beside it. Edit opens the check with the same fields and a Save check button. Remove takes the check out at once, with no confirmation. Nothing is saved until you select Save changes, so Cancel discards your changes. Add check takes a Weight from 1 to 100.
With no checks left, Save changes isn't available.
Faithfulness
Faithfulness rates how much of what the agent said is supported by the context it was given: the retrieved context for a turn, the results of the tools it called, and what the user said in the conversation. Nothing else counts as support.
The judge breaks each agent response into its individual factual claims. A claim is supported if the context states it or directly entails it. A claim is unsupported if the context contradicts it or is silent on it. The score is the number of supported claims divided by the total number of factual claims, shown in Online Eval as a percentage.
It judges support, not usefulness or correctness. A claim that is true but absent from the context is unsupported.
Some replies aren't counted either way. Questions, clarifications, refusals, and statements about what the agent itself is doing carry no factual claim. If the agent correctly says it doesn't know something, that is supported.
A turn with no retrieved context and no tool results isn't scored as a failure on that basis alone. The judge evaluates whatever the user supplied and reports a lower certainty.
Agent judge
With the Agent framework, one of your agents scores each output instead of a model. In the Evaluators table, the evaluator's Type reads Agent, its Method is a dash, and Scored by names the agent. Its runs appear in Online Eval Runs like any other evaluator's.
Agent
Required. The agent that scores.
Pass threshold
A percentage. Below this score an output fails.
Confidence below
Under Review when the judge is unsure. Any verdict below this percentage goes to a person. Failed executions always go to review.
The agent must reply in a fixed JSON format, so set the agent's response format to JSON. The dialog asks for JSON "so every reply can be scored."
Only evidence is optional, with at most three points.
Messages
Weight can be at most 50
The weight is above the limit for a new check.
Weight must be at least 1
The weight is below the limit.
No checks yet
The evaluator has no checks.
Add at least one check
A checklist evaluator needs a check before you can save.
Pick 2–5 models
A jury needs two to five models.
Choose which agents "<evaluator name>" scores.
Shown in the Select agents window.
Agent assignment updated
The agent assignment saved.
What's next
Last updated
Was this helpful?

