> For the complete documentation index, see [llms.txt](https://docs.uptiq.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.uptiq.ai/control-center/evals/evaluators.md).

# Evaluators

Create scoring methods and configure confidence-based review routing.

An evaluator is a named scoring method backed by a judge model, defined **at the account level**. Open **Evaluation & Quality → Evaluators** to view, edit, or create evaluators. Use **Agent enablement** to attach an account's evaluators to individual agents — an evaluator isn't created per agent or per project, so the same one can serve every agent that enables it.

At the top, the page summarizes your setup:

* Total and active evaluators
* Available judge models from Anthropic, OpenAI, and Google
* Agents with Online Eval enabled

Use **Evaluators** to manage scoring methods. Use **Agent enablement** to enable evaluators for each agent.

<figure><img src="https://1326225582-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F0qmgQjJ5aArDTj2ACFHG%2Fuploads%2Fgit-blob-4f0844303d66eec6928dd9b97b927b5dce4b6cc6%2Fqore_control-center_control-centre-evaluation-quality-evaluators_rounded_shadow.png?alt=media" alt="The Evaluators screen lists evaluators with their framework, judge model, confidence threshold, and status."><figcaption><p>Evaluators are the named scoring methods available to your agents.</p></figcaption></figure>

### Evaluator fields

| Field              | What it shows                                                                                   |
| ------------------ | ----------------------------------------------------------------------------------------------- |
| **Evaluator**      | The evaluator name, method, and number of agents using it.                                      |
| **Framework**      | The scoring approach: **Single judge** (one model) or **Jury** (several models, weighted vote). |
| **Judge model(s)** | The model that performs scoring, such as `gpt-5.1`.                                             |
| **Conf. → HITL**   | The confidence threshold that routes results to Human Review, such as `< 67%`.                  |
| **Status**         | Whether the evaluator is active.                                                                |
| **Actions**        | A menu for editing or removing the evaluator.                                                   |

### Create an evaluator

{% stepper %}
{% step %}

#### Open the creation dialog

Select **+ New evaluator**.
{% endstep %}

{% step %}

#### Configure the evaluator

Complete the following fields:

* **Evaluator name** — Required. Use a label such as *Answer relevancy*.
* **Framework** — **Single judge** scores each output with one model. **Jury** scores it with several judge models and takes a weighted vote. See [Choose a framework](#choose-a-framework) below.
* **Evaluation method** — Select **Answer Relevancy**, **Safety**, or **Task Completion**.

<figure><img src="https://1326225582-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F0qmgQjJ5aArDTj2ACFHG%2Fuploads%2Fgit-blob-3b5bc81c59fa3bd322e493e9591afbc99a0e01a8%2Fqore_control-center_control-centre-evaluation-quality-new-evaluator_rounded_shadow.png?alt=media" alt="The New Evaluator dialog shows fields for evaluator name, framework, and evaluation method."><figcaption><p>Configure a new evaluator.</p></figcaption></figure>

<figure><img src="https://1326225582-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F0qmgQjJ5aArDTj2ACFHG%2Fuploads%2Fgit-blob-58b978d4e9d503a44722d52d0f065dd79510c89a%2Fqore_control-center_control-centre-evaluation-quality-evaluation-method_rounded_shadow.png?alt=media" alt="The Evaluation method menu shows Answer Relevancy, Safety, and Task Completion."><figcaption><p>Select an evaluation method.</p></figcaption></figure>

* **Judge model** (Single judge) or **Jury models** (Jury) — Required. Select one model, or several for a jury.
* **Confidence threshold** — Defaults to `50%`. The judge (or jury) returns a rating and a confidence from 0–100%. Results below this threshold are flagged for human review, regardless of rating.
* **Route low-confidence results to human review** — Controls whether below-threshold results enter Human Review.
  {% endstep %}

{% step %}

#### Save the evaluator

Select **Create evaluator**. This button is enabled when you set a name, method, and at least one judge model.
{% endstep %}
{% endstepper %}

### Choose a framework

| Framework        | Judge models   | Scoring                                                                                                                           | Best for                                                                                            |
| ---------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| **Single judge** | One model      | That model scores each output directly.                                                                                           | Fast, low-cost scoring where one capable model's judgment is enough.                                |
| **Jury**         | Several models | Each judge model scores independently and reports its own confidence (0–100%). Qore combines the votes into one weighted verdict. | Higher-stakes evaluation methods, where no single model's judgment should decide the outcome alone. |

<figure><img src="https://1326225582-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F0qmgQjJ5aArDTj2ACFHG%2Fuploads%2Fgit-blob-d60e2cdc5bd2f048e5819b1750ab8923efa7dc43%2Fqore_control-center_control-centre-evaluation-quality-edit-evaluator-jury_rounded_shadow.png?alt=media" alt="The Edit evaluator dialog with Jury selected, showing five jury models (Claude Haiku 4.5, Claude Sonnet 4.5, GPT 5.1, Gemini 3.1 Pro Preview, GPT 4o) and a 95% confidence threshold"><figcaption><p>A Jury evaluator with five judge models and its own confidence threshold.</p></figcaption></figure>

<figure><img src="https://1326225582-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F0qmgQjJ5aArDTj2ACFHG%2Fuploads%2Fgit-blob-c2d9a79aa4a4ecb7ee6661f1cef4941e3cf8ead5%2Fqore_control-center_control-centre-evaluation-quality-edit-evaluator-single-judge_rounded_shadow.png?alt=media" alt="The Edit evaluator dialog with Single judge selected, showing one judge model and a 90% confidence threshold"><figcaption><p>A Single judge evaluator, for comparison.</p></figcaption></figure>

{% hint style="info" %}
A Jury evaluator's confidence threshold is independent of every other evaluator's. In the table above, **Jury Answer Relevancy** routes below `95%` while a **Single judge** evaluator on the same page can route below `67%` or `100%`. Set each evaluator's threshold for its own judge model(s), not to match its neighbors.
{% endhint %}

## Agent enablement

Use the **Agent enablement** tab to turn on Online Eval for an agent and choose which evaluators score its live traffic.

<figure><img src="https://1326225582-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F0qmgQjJ5aArDTj2ACFHG%2Fuploads%2Fgit-blob-c106b34827658e625dbc00e3519891f621a275bf%2Fqore_control-center_control-centre-evaluation-quality-agent-enablement_rounded_shadow.png?alt=media" alt="The Agent enablement tab, listing agents with their assigned evaluators, a sampling percentage, and an Online eval toggle"><figcaption><p>Turn on Online Eval per agent and choose its evaluators.</p></figcaption></figure>

Each row shows:

| Field                   | What it shows                                                                                                                                                  |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Agent**               | The agent this row configures.                                                                                                                                 |
| **Assigned evaluators** | The evaluators enabled for this agent, as chips. Select **Edit** to change them.                                                                               |
| **Sampling**            | The percentage of the agent's live traffic that runs through its assigned evaluators. Defaults to `100%`; lower it to control eval cost on high-volume agents. |
| **Online eval**         | The toggle that turns scoring on or off for this agent. Every enabled evaluator runs on the sampled responses.                                                 |

{% hint style="info" %}
An agent can show **Unsupported** instead of a sampling control and toggle. That agent's type doesn't support Online Eval yet — its **Edit** link still lets you pre-assign evaluators for when support ships.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.uptiq.ai/control-center/evals/evaluators.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.