> For the complete documentation index, see [llms.txt](https://docs.uptiq.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/online-eval-runs.md).

# Online Eval Runs

See every scored live run, how the judges scored it, and which runs went to review.

Online Eval scores live production traffic. Use it to check how your agents perform in production, find the runs that failed, and open any run to read the judge's reasoning and the conversation behind it. Open it in **Control Center → Evals → Online Eval Runs**.

The list covers the whole account. Use **Project** to narrow it to one project. A conversation can appear on several rows, one for each evaluator assigned to its agent. The **Scores** tab lists every scored run, and the **Runs** tab lists every run, including those with no score. See [Runs tab](#runs-tab).

![](https://1326225582-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F0qmgQjJ5aArDTj2ACFHG%2Fuploads%2Fgit-blob-bea1d65242f583c48078f0f9551b417e16ca3763%2Fonline-runs.png?alt=media)

## Permissions

| Action                                    | Requires                                                               |
| ----------------------------------------- | ---------------------------------------------------------------------- |
| Start a pending run with **Post trigger** | The **Build Evals** capability. The action isn't available without it. |

## Summary

The counts above the table summarize your runs.

| Count               | Shows                                                                          |
| ------------------- | ------------------------------------------------------------------------------ |
| **Scored**          | How many runs were scored, with how many passed and failed.                    |
| **Pass rate**       | The share of runs that scored at or above the pass mark with no Critical miss. |
| **Sent to review**  | How many runs went to a person, with their share of the scored runs.           |
| **Avg. confidence** | The average confidence of the judges, with the average score.                  |

## Filters

| Filter                    | Behavior                                                               |
| ------------------------- | ---------------------------------------------------------------------- |
| **Project** and **Agent** | Choosing an agent also sets the agent's project.                       |
| **Evaluator**             | A searchable list of every evaluator in the account.                   |
| **Outcome**               | **Passed**, **Failed**, or **In review**.                              |
| **Search**                | Type part of an execution ID or an evaluator name and press **Enter**. |

The filters are saved in the page address, so you can share a filtered view. The counts recalculate for **Project**, **Agent**, **Evaluator**, and **Search**.

## Run fields

| Column           | What it shows                                                                                                                                                                                                                                                          |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Execution ID** | The execution that was scored.                                                                                                                                                                                                                                         |
| **Agent**        | The agent that produced the output.                                                                                                                                                                                                                                    |
| **Evaluator**    | The evaluator that scored it.                                                                                                                                                                                                                                          |
| **Score**        | The result as a percentage, or `pass` or `fail` for a method that returns only pass or fail.                                                                                                                                                                           |
| **Confidence**   | How sure the judge is of its verdict.                                                                                                                                                                                                                                  |
| **Verdict**      | **Passed** or **Failed**.                                                                                                                                                                                                                                              |
| **Human review** | **Needs review** when the run was sent to the [Review Queue](https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/review-queue.md). A dash means it wasn't. The column marks that the run was routed, and it doesn't change when a reviewer records a verdict. |
| **Scored At**    | When the score was recorded.                                                                                                                                                                                                                                           |

## Review routing

A run goes to the [Review Queue](https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/review-queue.md) when the evaluation failed, when a critical check was missed, or when the judge's confidence is below the evaluator's review rule. The last case can happen even when the verdict is Passed. Review never changes the Pass or Fail result.

Routing applies to every evaluator that routes to review. An evaluator created earlier can have routing turned off. Its **Review rule** reads **Off** on the [Evaluators](https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/evaluators.md#evaluators-table) page, and its runs never show **Needs review**.

## Open a run

Select a row. The dialog is headed by the execution ID and has three tabs.

| Tab              | Shows                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Overview**     | The **Score** with the pass mark, and the **Judge confidence** with the evaluator's review rule. Below them are the **Verdict**, **Agent**, **Evaluator**, **Execution ID**, **Conversation ID**, when it was **Scored**, and whether **Human review** is needed. The last block explains the routing. It reads **Why the verdict is final** when no review is needed, and **Why a person reviews it** when the run goes to review. For a run that passed but has low confidence, it lists both the score line and the confidence line. |
| **Reasoning**    | One row for each judge, with its **Judge**, **Verdict**, **Score**, **Confidence**, and written **Reasoning**. The **Judge** column shows the name of the judge model. A long **Reasoning** is cut off with an ellipsis. Open a judge row's **Actions** menu and select **View details** to read it in full. A **Single judge** evaluator has one row. A **Jury** evaluator has one row for each juror, under a heading that counts how many jurors passed. The score and confidence on **Overview** combine the jurors' results.       |
| **Conversation** | The exchange the judge scored: the **User** message and the **Agent** reply, then **Tools used**. Each tool call shows whether it succeeded, its arguments, and what it returned. **Show more** expands a long result.                                                                                                                                                                                                                                                                                                                  |

## Judge evidence

**View details** on a judge row shows the judge's **Verdict**, **Score**, **Confidence**, and **Reasoning**, and below them **Evidence**. **Evidence** has up to three points, ordered from the one that moved the score most to the one that moved it least. Each point cites something specific the agent did or failed to do, and says whether it raised or lowered the score. A judge can return fewer points, or none, and then no **Evidence** list is shown.

## Runs tab <a href="#runs-tab" id="runs-tab"></a>

When an agent's live conversation is sampled, Qore creates one run for each evaluator assigned to that agent. A run that scores appears on **Scores**. A run that fails or is skipped has no score, so **Runs** is the only place it appears.

Filter by **Project**, **Agent**, or **Status**, and use **Search** to find an execution. Each row shows the **Execution**, **Agent**, **Evaluator**, **Status**, when the run was **Created**, and an **Actions** menu.

| Status        | Meaning                                                                                                                                                              |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Pending**   | The run is waiting to start. The **Actions** menu of a Pending run has **Post trigger**, which starts the run now. The run moves to **Running**, then **Completed**. |
| **Running**   | The judge is scoring the conversation.                                                                                                                               |
| **Completed** | The run produced a score, which appears on **Scores**.                                                                                                               |
| **Failed**    | The run ended without a score. The status gives a reason. See [Failure reasons](#failure-reasons).                                                                   |
| **Skipped**   | The run was never scored, either because its evaluator was deleted before scoring started or because the conversation had nothing to score.                          |

### Failure reasons

| Reason                                          | Meaning                                                                    |
| ----------------------------------------------- | -------------------------------------------------------------------------- |
| `Judge model unavailable: <model>`              | The evaluator uses a judge model that isn't available.                     |
| `Evaluation method not found`                   | The evaluator's method no longer exists.                                   |
| `Evaluation timed out`                          | The run stayed **Pending** or **Running** for more than two hours.         |
| `Judge response was not in the expected format` | The judge answered, but its answer couldn't be read as a score.            |
| `Evaluator agent not found`                     | The evaluator uses an agent as its judge, and that agent no longer exists. |
| `Evaluator agent run did not complete`          | The agent that scores the conversation didn't finish.                      |
| `Evaluation failed`                             | Scoring hit an error.                                                      |

## Export results

Online Eval has no export button of its own. To download evaluation results, open **Control Center → Observe → Export** and choose the **Eval Results** dataset: scores, verdicts, reasoning, evidence, and human overrides for evaluated agent runs.

The Export page sets the project scope, date range, format, and delivery, and can run an export once or on a schedule. Large exports run in the background. See [Export](https://docs.uptiq.ai/ai-workbench-qore/control-center/observe/export.md).

## What the judge scores

Each evaluator scores a transcript of the conversation.

| Transcript part | Holds                                                                   |
| --------------- | ----------------------------------------------------------------------- |
| Messages        | Every message, with its formatting kept inside the turn it belongs to.  |
| Documents       | The documents the user attached and the agent produced, on each turn.   |
| Tool calls      | Each tool call's arguments, whether it succeeded, and what it returned. |
| Rich replies    | Replies such as widgets, rendered as text.                              |

Long tool results are shortened, and the transcript says where content was cut. A conversation too long for the judge model is trimmed to whole turns, and the transcript notes how many turns were left out. A judge sees what the tools returned, so an answer built on a failed tool call can score differently from the same answer built on a successful one.

## Messages

| Message                                                            | Meaning                           |
| ------------------------------------------------------------------ | --------------------------------- |
| "Evaluation is running in background, please check after sometime" | **Post trigger** started the run. |

## What's next

<table data-view="cards"><thead><tr><th>Title</th><th>Description</th><th data-card-target data-type="content-ref">Link</th></tr></thead><tbody><tr><td><strong>Review Queue</strong></td><td>Check the runs a person needs to review and record a verdict.</td><td><a href="https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/review-queue.md">Review Queue</a></td></tr><tr><td><strong>Reviewer Feedback</strong></td><td>Read what reviewers say should change in an agent.</td><td><a href="https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/reviewer-feedback.md">Reviewer Feedback</a></td></tr><tr><td><strong>Evaluators</strong></td><td>Define how outputs are scored and who reviews them.</td><td><a href="https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/evaluators.md">Evaluators</a></td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/online-eval-runs.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
