Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Evaluators

Create scoring methods, set a pass mark, and decide which results a person reviews.

Evaluators are named, account-level scoring methods backed by a judge model. They help you apply consistent quality standards across agents, set a pass mark, and route uncertain or failed results for review.

Open Control Center → Evals → Evaluators to view, edit, or create evaluators. An evaluator isn't created per agent or per project, so the same one can serve every agent that enables it. Attach an evaluator to individual agents in Agent List. For task-focused instructions, see Create an evaluator in Task Guides.

Permissions

Action
Requires

+ New evaluator

The Build Evals capability. Without it, the button isn't available.

Evaluators table

The table lists your evaluators, with Search and a + New evaluator button. Type part of an evaluator's name in Search and press Enter to narrow the table.

Column
What it shows

Evaluator

The evaluator name.

Type

What the evaluator returns, or who scores: Checklist, Single score, Pass or fail, or Agent.

Method

The evaluation method, such as Answer Relevancy. A dash for an evaluator that an agent scores.

Scored by

The framework and what scores: Single judge with the name of its model, Jury with the names of its models, or Agent with the agent's name.

Pass mark

The lowest score that passes, shown with ≥ and a percentage. Methods that return only pass or fail show Pass / fail.

Review rule

Which results a person also checks, shown as Confidence < and a percentage, or Off for an evaluator whose routing to the Review Queue is turned off.

Agents

How many agents the evaluator scores.

Status

Active for an evaluator that is in use.

Actions

A menu with Edit and Assign agents.

An evaluator only scores the agents assigned to it. Assigning alone doesn't start scoring: the agent's Online eval switch must be on too. See Agent List.

Evaluation methods

Choose a method based on the quality signal you need. The method determines what the judge evaluates and whether it returns a score or a pass-or-fail result.

Method
The judge determines
Returns

Checklist

Whether the agent satisfied each checklist item you write. Items can be weighted and marked as critical.

A result per check, combined into a weighted score.

Answer Relevancy

How much of the agent's answer addresses what the user asked.

Single score

Faithfulness

How much of the agent's answer is supported by the context it received.

Single score

Safety

Whether the agent avoided harmful content and kept personal data out of its responses.

Pass or fail

Task Completion

Whether the agent accomplished what the user was trying to do.

Single score

Frameworks

Framework
Judge models
Scoring
Best for

Single judge

One model

That model scores each output directly.

Fast, low-cost scoring where one capable model's judgment is enough.

Jury

Several models

Each judge model scores independently and reports its own confidence as a percentage from 0 to 100. Qore combines the votes into one weighted verdict.

Higher-stakes evaluation methods, where no single model's judgment should decide the outcome alone.

Agent

One of your agents

The agent scores each output and replies with a score, a confidence, and its reasoning. See Agent judge.

Scoring that follows an agent you have built.

The Edit evaluator dialog with Jury selected, showing a 95% confidence threshold and five jury models: Claude Haiku 4.5, Claude Sonnet 4.5, GPT 5.1, Gemini 3.1 Pro Preview, and GPT 4o
A Jury evaluator with five judge models and its own confidence threshold.
The Edit evaluator dialog with Single judge selected, showing one judge model and a 90% confidence threshold
A Single judge evaluator, for comparison.

An evaluator's confidence threshold is independent of every other evaluator's, so one evaluator can route below one percentage while another routes below a different one. Set each evaluator's threshold for its own judge model or models, not to match its neighbors.

Evaluator configuration

The New Evaluator dialog has five steps, shown as tabs: Method, Basics, a step named for the method, Gates and routing, and Agents. The method step is Checklist for a checklist and Method details for the other methods. Back and Next move between them.

Method

Method reads "Start with how it should judge. Everything else comes with sensible defaults." Each method is a card with a one-line description. Checklist is first, marked Recommended, and selected by default.

Basics

Basics identifies the evaluator and its judge. Next isn't available until you enter an Evaluator name.

Field
Reference

Evaluator name

Required.

Framework

Single judge uses one model. Jury uses several judge models and takes a weighted vote.

Judge model

Required for Single judge.

Jury models

Required for Jury. Select 2 to 5 models. Models can come from different providers. The picker is a searchable list with provider filters, and it starts with one model selected. It displays "Pick 2–5 models," and Next isn't available until you select two to five.

Method details

Method details describes what a non-checklist method asks and returns. The platform manages these values, so you change nothing here.

Field
Reference

Judge question

The question used to evaluate the response. For Task Completion: "Did the agent complete the task it was given?"

Output

A Single score from 0 to 100, with a written reason.

Confidence

One figure for the whole verdict.

Prompt

Managed by the platform.

Switch to Checklist gives each check its own result, with weights and Critical checks.

Gates and routing

Gates and routing reads "When an output passes, and when a person reviews it." A person always checks every failed output, every Critical miss, and any output the judge is unsure about.

Field
Reference

Pass mark

Pass at or above a percentage. Below this score, an output fails. A pass-or-fail method has no score to set.

Human review

Confidence below a percentage, or Juror agreement below for a jury. Any verdict below this is reviewed. In Edit, a jury's control is labeled Confidence below.

Review never changes the Pass or Fail result.

Confidence is how sure the judge is of its verdict. For a checklist, it is the lowest confidence across the checks. For a jury, it is how many jurors agree with the majority.

Agents

Agents lists each agent with its project and evaluator count. Select all and Clear change the selection. You can also assign agents later. The button reads Create evaluator. After you select agents, it changes to, for example, Create and assign to 1 agent, and saving assigns them.

Review routing

An evaluator you create in the dialog always routes to Review Queue. Evaluators created earlier can have routing turned off. Their Review rule reads Off, and none of their results go to the Review Queue.

In Edit, Review when the judge is unsure is a switch. Turn it off to stop routing for the evaluator. The Confidence below control disappears, and the Review rule reads Off. Pass threshold and Confidence below each have a slider and a number box. Under them, the dialog notes that failed executions always go to human review, and, for a checklist, critical check misses too.

Checklist

Checklist scores whether the agent satisfied each checklist item you write, with weighted and critical items. An evaluator can have up to 10 checks, and at least one.

Field
Reference

Check

A name for the check.

What the judge looks for

The criterion, written as a statement the judge can verify.

Weight

How much the check counts. Weights are scaled to 100% across the checks that apply, so a weight of 30 next to one of 20 gives shares of 60% and 40%. The default is 10.

Critical

A switch. Missing a critical check fails the output outright, whatever the score, and sends it to a person.

The judge rules on every check. The score is the total weight of the checks that passed divided by the total weight of all checks. The output passes only when the score reaches the pass mark and no critical check was missed. A missed critical check fails the evaluation whatever the score, and it always goes to Review Queue.

Checks in the dialog

Place
Behavior

New Evaluator, Checklist step

Counts your checks and your Critical checks, and draws a bar of their shares. The table lists each check with its Share. A check's checkbox turns it on or off. When you clear it, the check is left out of scoring, its Share reads a dash, and the other checks take its share. A row's Actions menu has Edit and Remove. Add check takes a Weight from 1 to 50.

Edit

Shows how many checks are turned on out of the total, with each check's Weight and a Critical badge, and Edit and Remove icons beside it. Edit opens the check with the same fields and a Save check button. Remove takes the check out at once, with no confirmation. Nothing is saved until you select Save changes, so Cancel discards your changes. Add check takes a Weight from 1 to 100.

With no checks left, Save changes isn't available.

Faithfulness

Faithfulness rates how much of what the agent said is supported by the context it was given: the retrieved context for a turn, the results of the tools it called, and what the user said in the conversation. Nothing else counts as support.

The judge breaks each agent response into its individual factual claims. A claim is supported if the context states it or directly entails it. A claim is unsupported if the context contradicts it or is silent on it. The score is the number of supported claims divided by the total number of factual claims, shown in Online Eval as a percentage.

  • It judges support, not usefulness or correctness. A claim that is true but absent from the context is unsupported.

  • Some replies aren't counted either way. Questions, clarifications, refusals, and statements about what the agent itself is doing carry no factual claim. If the agent correctly says it doesn't know something, that is supported.

  • A turn with no retrieved context and no tool results isn't scored as a failure on that basis alone. The judge evaluates whatever the user supplied and reports a lower certainty.

Agent judge

With the Agent framework, one of your agents scores each output instead of a model. In the Evaluators table, the evaluator's Type reads Agent, its Method is a dash, and Scored by names the agent. Its runs appear in Online Eval Runs like any other evaluator's.

Field
Reference

Agent

Required. The agent that scores.

Pass threshold

A percentage. Below this score an output fails.

Confidence below

Under Review when the judge is unsure. Any verdict below this percentage goes to a person. Failed executions always go to review.

The agent must reply in a fixed JSON format, so set the agent's response format to JSON. The dialog asks for JSON "so every reply can be scored."

Only evidence is optional, with at most three points.

The New Evaluator dialog offers Single judge and Jury. An evaluator's framework and method are fixed once it exists. In Edit, the other frameworks aren't available.

Messages

Message
Meaning

Weight can be at most 50

The weight is above the limit for a new check.

Weight must be at least 1

The weight is below the limit.

No checks yet

The evaluator has no checks.

Add at least one check

A checklist evaluator needs a check before you can save.

Pick 2–5 models

A jury needs two to five models.

Choose which agents "<evaluator name>" scores.

Shown in the Select agents window.

Agent assignment updated

The agent assignment saved.

What's next

Last updated

Was this helpful?