> For the complete documentation index, see [llms.txt](https://docs.uptiq.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.uptiq.ai/agent-builder/test/evals.md).

# Evals

Evals is an evaluation framework for systematically testing agent quality on the Uptiq Platform. It allows teams to create datasets of test cases — manually or auto-generated by an LLM — run those test cases against a live agent, score each response on a 0–10 scale using an LLM scorer, and receive AI-generated suggestions for improving the agent.

{% hint style="info" %}
Use [Evals](/agent-builder/test/evals.md) before release, [Online-Eval](/agent-builder/monitor/online-eval.md) after release, and [Reinforcement Learnings](/agent-builder/monitor/reinforcement-learnings.md) to carry reviewed feedback forward.
{% endhint %}

| Capability           | Description                                                                         | Limit             |
| -------------------- | ----------------------------------------------------------------------------------- | ----------------- |
| Dataset management   | Create, rename, delete datasets; auto-generate test cases from agent config via LLM | 100 per agent     |
| Test case management | Question, expected answer, optional document context and attachments                | 50 per generation |
| Evaluation runs      | Trigger, monitor progress in real-time, cancel, view results                        | 100 per agent     |
| LLM scoring          | 0–10 scale with reasoning, 7+ = pass threshold                                      | —                 |
| Suggested changes    | AI-generated improvement recommendations with one-click apply                       | —                 |
| Document support     | Auto-generated DOCX/XLSX/PDF test documents with preview                            | —                 |

Evals is accessed from the **Cognition & Control** section in the Agent Builder's **Manage Agent** sidebar.

## Workflow

{% tabs %}
{% tab title="Datasets" %}
Manage [datasets](#datasets) and [test cases](#test-cases).
{% endtab %}

{% tab title="Runs" %}
View [runs](#runs), [scoring](#scoring), and [suggested changes](#suggested-changes).
{% endtab %}
{% endtabs %}

{% stepper %}
{% step %}

### Build a dataset

Create a [dataset](#datasets) manually or generate one with AI.
{% endstep %}

{% step %}

### Run the evaluation

Execute the dataset against the live agent in [Runs](#runs).
{% endstep %}

{% step %}

### Review and improve

Inspect [scores](#scoring), review failures, and apply [suggested changes](#suggested-changes).
{% endstep %}
{% endstepper %}

***

## Datasets

A dataset is a collection of test cases for evaluating an agent. Each agent can have up to **100 datasets**.

### Generate Dataset

{% tabs %}
{% tab title="Generate with AI" %}
Users click **"Generate Dataset"** to auto-generate test cases.

| Field                       | Required | Description                                    |
| --------------------------- | -------- | ---------------------------------------------- |
| **Dataset Name**            | Yes      | Defaults to "Eval Dataset {date}"              |
| **Number of Test Cases**    | Yes      | 1–50, defaults to 10                           |
| **Additional Instructions** | No       | e.g., "Focus on edge cases related to pricing" |

**Generation steps:**

1. The system reads the agent's system prompt (instructions)
2. Sends it to an LLM (Gemini 3 Flash) to generate test cases
3. For each test case, the LLM creates: question, expected answer, and optional document content
4. Document content is converted to actual files (DOCX, XLSX, or PDF) and uploaded to storage
5. Test case records are created with document references
   {% endtab %}

{% tab title="Add manually" %}
Create manual test cases one at a time in an existing dataset.

Each manual test case can include:

* `question`
* `expected answer`
* optional documents

Use `POST /:agentId/eval/datasets/:datasetId/items` to create the test case.

Use **Test Case Search** to filter by question, expected answer, or document context.
{% endtab %}
{% endtabs %}

<details>

<summary>How document format selection works</summary>

Document format selection is automatic.

| Format | Content Type             | Examples                      |
| ------ | ------------------------ | ----------------------------- |
| XLSX   | Tabular/spreadsheet data | Budgets, financial tables     |
| DOCX   | Narrative prose          | Memos, policies, contracts    |
| PDF    | Professional documents   | Invoices, resumes, statements |

</details>

### Dataset Views

**List view** displays all datasets for the agent:

* **Name** — Dataset name
* **Created** — Creation timestamp
* **Actions** — Rename, Delete

Empty state: "No evaluation datasets yet. Generate one to get started."

**Details view** shows the dataset name as header with a **"Run Evaluation"** button (disabled if no [test cases](#test-cases)), plus the list of all [test cases](#test-cases).

***

### Test Cases

Each test case contains:

* **Question** — The input prompt sent to the agent
* **Expected Answer** — The ground truth / ideal response
* **Document Context** (optional) — Description of attached documents (e.g., "Excel budget sheet")
* **Documents** (optional) — Attached files (DOCX, XLSX, PDF) shown as clickable chips

#### Document Preview

Clicking a document chip opens a preview modal:

| File Type                  | Viewer                                  |
| -------------------------- | --------------------------------------- |
| PDF                        | Native document viewer                  |
| Google Docs / Office files | Embedded Google or Office Online viewer |
| Excel                      | Office Online viewer                    |

{% hint style="info" %}
Toggle between "Show preview" and "Show context" if document context is available. Use the "Open file" button for external download.
{% endhint %}

***

## Runs

#### Starting a Run

{% tabs %}
{% tab title="Start a run" %}

1. Navigate to a dataset with at least one test case
2. Click **"Run Evaluation"**
3. The system creates an eval run with status `Pending`
4. Background processing starts
5. Success toast: "Evaluation run started"
   {% endtab %}

{% tab title="Monitor a run" %}
Use the **Runs** tab to track:

* current status
* progress
* scores
* suggestions
  {% endtab %}
  {% endtabs %}

#### Run Execution

For each test case in the dataset:

1. Resolve document references to signed URLs
2. Trigger the agent via the Agent Executor (`/agents/{agentId}/trigger`)
3. Pass the question + documents to the agent
4. Extract the agent's actual answer from the response
5. Score the response using an LLM scorer (Gemini 3 Flash)
6. Record the result (score, reasoning, correctness, duration)
7. Update run progress counters

{% hint style="info" %}
Up to **3 test cases** execute concurrently during a run.
{% endhint %}

#### Run Status

<details>

<summary>Status lifecycle</summary>

```
Pending → Running → Completed
                 → Failed
                 → Cancelled (user-stopped)
```

</details>

| Status        | Indicator        | Details Shown                               | Available Actions                                     |
| ------------- | ---------------- | ------------------------------------------- | ----------------------------------------------------- |
| **Pending**   | Warning badge    | Waiting to start                            | —                                                     |
| **Running**   | Animated spinner | Progress bar (completedItems / totalItems)  | Stop                                                  |
| **Completed** | Green badge      | Pass/fail counts, [average score](#scoring) | View results, Apply [suggestions](#suggested-changes) |
| **Failed**    | Red badge        | Error message                               | —                                                     |
| **Cancelled** | Red badge        | Partial [results](#scoring)                 | View partial [results](#scoring)                      |

{% hint style="info" %}
A **Stop** button appears during in-progress runs. Stopping preserves already-completed results. The UI polls every **3 seconds** while a run is in progress, stopping automatically when the run completes.
{% endhint %}

***

## Scoring

**Score Scale:** Each test case response is scored by an LLM (Gemini 3 Flash):

| Score    | Verdict           | Description                                          | Result |
| -------- | ----------------- | ---------------------------------------------------- | ------ |
| **0–3**  | Incorrect         | Response is wrong or irrelevant                      | Fail   |
| **4–6**  | Partially correct | Some relevant content but incomplete or inaccurate   | Fail   |
| **7–8**  | Correct           | Meets the expected answer criteria                   | Pass   |
| **9–10** | Comprehensive     | Exceeds expectations with additional relevant detail | Pass   |

{% hint style="warning" %}
**Pass threshold:** A score of 7 or above marks the test case as passed.
{% endhint %}

**Result Card** — each completed test case displays:

* **Status Badge** — Pass (green), Fail (red), or Error (red)
* **Score** — "Score: X/10"
* **Duration** — Execution time
* **Question** — Original input
* **Expected Answer** — Ground truth (muted, scrollable)
* **Actual Answer** — Agent's response (or error message)
* **Scorer Reasoning** — LLM's explanation of the score

Results appear progressively as each test case completes.

{% hint style="success" %}
Completed runs display **pass count** (green), **fail count** (red), **average score** (0–10), and an LLM-generated **summary** of evaluation results.
{% endhint %}

<details>

<summary>What a failing run usually means</summary>

A failing run does not always mean the agent is broken.

Common causes include:

* incomplete instructions
* missing skills or knowledge
* answers that are directionally correct but below the pass threshold

</details>

***

## Suggested Changes

{% hint style="info" %}
Suggestions are generated automatically when a run completes with failures or an average score below 7.
{% endhint %}

Each suggestion contains:

* **Title** — Short description (max 200 characters)
* **Category** — Instructions, Skills, Knowledge, Guardrails, or Other
* **Description** — Detailed recommendation (max 2,000 characters)

Suggestions can modify: agent instructions, skills, knowledge, and guardrails.

#### Applying Suggestions

{% stepper %}
{% step %}

### Open suggestions

Click **"Apply to Agent"** on a completed run with suggestions.
{% endstep %}

{% step %}

### Review selections

A dialog opens with all suggestions and checkboxes.

Edit titles, categories, or descriptions as needed.
{% endstep %}

{% step %}

### Apply changes

Select the suggestions to apply and click **Apply**.

Toast: "Builder agent is applying your changes — watch the chat panel on the left"
{% endstep %}

{% step %}

### Continue in chat

The agent config modal closes and the Builder Agent executes the changes in the chat panel.
{% endstep %}
{% endstepper %}

### Related sections

* [Monitoring](broken://pages/cD46XWa7ZwMLNvtnC3yG) — Watch runtime behavior after the agent is live.
* [Online-Eval](/agent-builder/monitor/online-eval.md) — Score live conversations in production.
* [Reinforcement Learnings](/agent-builder/monitor/reinforcement-learnings.md) — Review and approve learning cases from real usage.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.uptiq.ai/agent-builder/test/evals.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.