> For the complete documentation index, see [llms.txt](https://docs.uptiq.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.uptiq.ai/governance/evaluation-and-quality.md).

# Evaluation & Quality

Judge whether agents produce good output. Define **evaluators**, monitor production executions in **Online Eval**, and route low-confidence cases to **Human Review**.

<figure><img src="/files/Q3PaGMuolu7lJxwjEa20" alt="The Evaluators screen, listing evaluators with framework, judge model, confidence threshold, and status"><figcaption><p>Evaluators, the named scoring methods available to your agents.</p></figcaption></figure>

{% columns %}
{% column %}

#### Configure quality checks

Create evaluators that score executions with a judge model and confidence threshold.
{% endcolumn %}

{% column %}

#### Review uncertain outcomes

Watch live results, then resolve tasks routed to a person.
{% endcolumn %}
{% endcolumns %}

#### At a glance

| Area             | Use it to                                     | Start here                    |
| ---------------- | --------------------------------------------- | ----------------------------- |
| **Evaluators**   | Define scoring methods and routing thresholds | [Evaluators](#evaluators)     |
| **Online Eval**  | Monitor scored production executions          | [Online Eval](#online-eval)   |
| **Human Review** | Resolve low-confidence review tasks           | [Human Review](#human-review) |

An evaluator scores each execution and reports its confidence. High-confidence results are accepted automatically. Low-confidence results route to Human Review for a person to decide.

## Evaluators

An evaluator is a named scoring method backed by a judge model. Open **Evaluation & Quality → Evaluators** to see the ones your account has and to create more.

At the top, the page summarizes your setup: how many evaluators exist and how many are active, how many judge models are available across Anthropic, OpenAI, and Google, and how many agents have online eval running. Two tabs sit below — **Evaluators**, the list, and **Agent enablement**, where you turn evaluators on for each agent.

Each evaluator is listed with:

| Field              | What it shows                                                                              |
| ------------------ | ------------------------------------------------------------------------------------------ |
| **Evaluator**      | The evaluator's name, its evaluation method, and how many agents use it.                   |
| **Framework**      | How it scores — **Single judge** (one model scores each output).                           |
| **Judge model(s)** | The model doing the scoring (for example, `gpt-5.1`).                                      |
| **Conf. → HITL**   | The confidence threshold below which results route to human review (for example, `< 67%`). |
| **Status**         | Whether the evaluator is **Active**.                                                       |
| **Actions**        | A per-row menu to edit or remove the evaluator.                                            |

### Create an evaluator

{% stepper %}
{% step %}

#### Open the creation dialog

Click **+ New evaluator**.
{% endstep %}

{% step %}

#### Configure the evaluator

Fill in the following fields:

* **Evaluator name** — required. A label for this evaluator (for example, *Answer relevancy*).
* **Framework** — **Single judge** scores each output with one model. **Jury vote** (several judges, weighted) is marked *Coming soon* and can't be selected yet.
* **Evaluation method** — the platform method to score against: **Answer Relevancy**, **Safety**, or **Task Completion**.

<figure><img src="/files/V1Cdgwh5QuUE1Cxm1Pjg" alt="The New Evaluator dialog, with evaluator name, framework, and evaluation method fields"><figcaption><p>Configuring a new evaluator.</p></figcaption></figure>

<figure><img src="/files/KFSWQtuWAFh0apTUuyqQ" alt="The Evaluation method dropdown, showing Answer Relevancy, Safety, and Task Completion options"><figcaption><p>Choosing the evaluation method to score against.</p></figcaption></figure>

* **Judge model** — required. The model that does the judging.
* **Confidence threshold** — a slider, defaulting to `50%`. The judge returns a rating together with a self-reported confidence from 0–100%. Any execution whose confidence falls below this threshold is flagged for human review regardless of its rating.
* **Route low-confidence results to human review** — a per-evaluator preference for sending below-threshold results to Human Review.
  {% endstep %}

{% step %}

#### Save the evaluator

Click **Create evaluator** to save. The button stays disabled until the required fields — name, evaluation method, and judge model — are set.
{% endstep %}
{% endstepper %}

## Online Eval

Online Eval scores live production traffic. Open **Evaluation & Quality → Online Eval** to see every execution an evaluator has scored and where it went.

<figure><img src="/files/r9J9TV7wfGJaMRsTN5Jz" alt="The Online Eval screen, listing scored executions with quality, confidence, verdict, and routing columns"><figcaption><p>Online Eval, scoring live production executions.</p></figcaption></figure>

You can narrow the view by agent, by minimum score, by verdict, or by routing, and search by execution, agent, or evaluator. Across the selected window you see how many executions were scored, how many **Passed** at or above the pass threshold, how many were **Routed to Human Review** below judge confidence, and the average score, with an **Open board** shortcut into Human Review.

Each row is one scored execution:

| Column        | What it shows                                                                                 |
| ------------- | --------------------------------------------------------------------------------------------- |
| **Execution** | The execution's ID.                                                                           |
| **Agent**     | Which agent produced the output.                                                              |
| **Evaluator** | Which evaluator scored it (for example, *PII Mask*, *Test valuator*).                         |
| **Qual.**     | The quality result — a `pass` or a percentage score.                                          |
| **Conf.**     | The judge's confidence, shown against the evaluator's threshold (for example, `0.96 ≥ 1.00`). |
| **Verdict**   | **Passed** or **Failed**.                                                                     |
| **Routing**   | **Auto-accepted** or **Human Review**.                                                        |

{% hint style="info" %}
**Routing depends on confidence, not the verdict**

An execution is routed to Human Review when the judge's confidence falls below the evaluator's threshold — even if its verdict is *Passed*.

Conversely, a *Failed* execution whose confidence is above threshold is *Auto-accepted*. Routing asks "is the judge sure?", not "did it pass?".
{% endhint %}

## Human Review

Human Review is the workspace where a person records a verdict on the executions that scored below an evaluator's confidence. Open **Evaluation & Quality → Human Review**. Tasks move from **Review Pending** (waiting on a reviewer's verdict) to **Reviewed** (verdict recorded).

<figure><img src="/files/oq5w77sfneuxTdKfvMmh" alt="The Human Review screen, listing tasks with team, assignee, priority, status, and verdict columns"><figcaption><p>Human Review, the queue of tasks awaiting a reviewer's verdict.</p></figcaption></figure>

Switch between **My tasks**, **All tasks**, and **Unassigned**, and narrow by team, type, status, or priority. Your queue is summarized as **Assigned to me**, **Reviews in view**, **Reviewed**, and **Unassigned**, with a **View unassigned** shortcut. Reviews are assigned through teams, so a reviewer who belongs to no team has nothing assigned to them.

Each task row shows the **Execution**, the **Eval type** and **Evaluator** that flagged it, the **Team** and **Assignee** responsible, its **Priority**, when it was **Queued**, its **Status**, and the recorded **Verdict**. Open a task to record your verdict and move it to *Reviewed*.

{% hint style="info" %}

### Notes

* **Single judge** is the only framework available today; **Jury vote** is *Coming soon*.
* **Confidence threshold** is set per evaluator, so different evaluators can route to human review at different confidence levels (`< 67%`, `< 100%`, and so on).
* **Human Review assignments** flow through **teams** — set up in [Account Settings](broken://pages/5d03a7284a20a0285053794e8d0475f4b80d483e#teams) — not to individuals directly.
  {% endhint %}

## Related

{% content-ref url="/spaces/0qmgQjJ5aArDTj2ACFHG/pages/807cab861e090edd0ce825909db3db1b28f73799" %}
[Control Center](/start-here/control-center.md)
{% endcontent-ref %}

{% content-ref url="/spaces/0qmgQjJ5aArDTj2ACFHG/pages/4023fee41d519b1fbe13cbf03c9027c75b45de1a" %}
[Observability](/governance/observability.md)
{% endcontent-ref %}

{% content-ref url="/spaces/0qmgQjJ5aArDTj2ACFHG/pages/049796530f3942b2e739e565a1e673ad4924290e" %}
[Controls & Audit](/governance/controls-and-audit.md)
{% endcontent-ref %}

{% content-ref url="/spaces/0qmgQjJ5aArDTj2ACFHG/pages/84115b130456683eaee9079209f1add7ef90110d" %}
[Account Settings](/governance/account-settings.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.uptiq.ai/governance/evaluation-and-quality.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.