> For the complete documentation index, see [llms.txt](https://docs.uptiq.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.uptiq.ai/ai-workbench-qore/task-guides/create-an-evaluator.md).

# Create an evaluator

Create an evaluator, change its checks and thresholds, and assign the agents it scores.

Create an evaluator that scores agent outputs, then assign it to the agents it should score. Each task below stands on its own.

## Before you begin

* Have the **Build Evals** capability. Without it, **+ New evaluator** isn't available.
* Decide the method, the judge model or jury models, and, for a checklist, the checks you want.
* To score with an agent instead of a model, set that agent's response format to JSON. See [Agent judge](https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/evaluators.md#agent-judge).

## Create an evaluator

1. Open **Control Center → Evals → Evaluators**.
2. Select **+ New evaluator**.
3. In **Method**, choose a method. **Checklist** is selected by default. Select **Next**.
4. In **Basics**, enter an **Evaluator name**.
5. Choose the **Framework**: **Single judge** or **Jury**.
6. Choose the **Judge model**, or for a jury, choose two to five **Jury models**. Select **Next**.
7. For a checklist, select **Add check** for each item. Enter a **Check**, **What the judge looks for**, and a **Weight**, and turn on **Critical** where a miss must fail the output. Select **Next**. For the other methods, review **Method details** and select **Next**.
8. In **Gates and routing**, set **Pass at or above** and **Confidence below**. For a jury, the second control is **Juror agreement below**. Select **Next**.
9. In **Agents**, select the agents this evaluator scores, or leave them empty to assign later.
10. Select **Create evaluator**.

The evaluator appears in the table. When you selected agents, the button reads, for example, **Create and assign to 1 agent**, and saving assigns them. A toast confirms with `Agent assignment updated`. Creating needs a name, a method, a judge model or jury models, and, for a checklist, at least one check. For what each field does, see [Evaluator configuration](https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/evaluators.md#evaluator-configuration).

## Assign agents to an evaluator

1. Open the evaluator's **Actions** menu.
2. Select **Assign agents**.
3. In **Select agents**, select **Search**, type part of an agent's name, and press **Enter** to narrow the list.
4. Select or clear the checkboxes. The agents that already use the evaluator are selected.
5. Select **Save selection**.

The toast `Agent assignment updated` appears, and the **Agents** count in the evaluator's row updates. Assigning alone doesn't start scoring. Turn on the agent's **Online eval** switch too. See [Agent List](https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/agent-list.md).

## Change an evaluator

1. Open the evaluator's **Actions** menu.
2. Select **Edit**.
3. Change the judge, the checks, the **Pass threshold** or **Confidence below**. To stop routing results to review, turn off **Review when the judge is unsure**.
4. Select **Save changes**.

The framework and method are fixed once the evaluator exists. For an evaluator scored by an agent, the dialog shows **Framework** as **Agent**, and you change the agent or its thresholds.

## Add or change a check

1. Open the evaluator's **Actions** menu, and select **Edit**.
2. Select **Add check**, or select the **Edit** icon beside a check.
3. Change the **Check**, **What the judge looks for**, **Weight** and **Critical** switch.
4. Select **Save check**.
5. Select **Save changes**.

**Remove** takes a check out at once, with no confirmation, but nothing is saved until you select **Save changes**. **Cancel** discards your changes. An evaluator needs at least one check and can have up to 10.

## What's next

<table data-view="cards"><thead><tr><th>Title</th><th>Description</th><th data-card-target data-type="content-ref">Link</th></tr></thead><tbody><tr><td><strong>Evaluators</strong></td><td>Reference for methods, frameworks, checks and thresholds.</td><td><a href="https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/evaluators.md">Evaluators</a></td></tr><tr><td><strong>Agent List</strong></td><td>Turn on online eval and set sampling rates for agents.</td><td><a href="https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/agent-list.md">Agent List</a></td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.uptiq.ai/ai-workbench-qore/task-guides/create-an-evaluator.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
