> For the complete documentation index, see [llms.txt](https://docs.uptiq.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.uptiq.ai/ai-workbench-qore/console/agent-builder/test/regression-evals.md).

# Regression Evals

Test agent responses against scored datasets with automated LLM grading.

Regression Evals is an evaluation framework for systematically testing agent quality on the Uptiq Platform. It allows teams to create datasets of test cases — read from an uploaded document, generated by an LLM, or added manually — run those test cases against a live agent, score each response on a 0–10 scale using an LLM scorer, and receive AI-generated suggestions for improving the agent.

{% hint style="info" %}
Use [Regression Evals](https://docs.uptiq.ai/ai-workbench-qore/console/agent-builder/test/regression-evals.md) before release, [Online Evals in Control Center](https://docs.uptiq.ai/ai-workbench-qore/control-center/evals.md) after release, and [Reinforcement Learnings](https://docs.uptiq.ai/ai-workbench-qore/console/agent-builder/monitor/reinforcement-learnings.md) to carry reviewed feedback forward.
{% endhint %}

<details>

<summary>Regression Evals or Evals? The difference</summary>

Qore has two evaluation features, and their names are close. **Regression Evals** is the one on this page. **Evals** is a separate section of Control Center. They answer different questions.

|                                  | **Regression Evals**                                                        | **Evals** (Control Center)                                                                                                                        |
| -------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Question it answers**          | Does the agent still give the right answers after I change it?              | How good are the answers the agent gives to real users?                                                                                           |
| **When it runs**                 | Before release, when you choose to run it                                   | After release, in the background on live conversations                                                                                            |
| **What is scored**               | The agent's replies to your test cases                                      | Real executions, which the agent ran for real users                                                                                               |
| **What you compare against**     | An expected answer you wrote, or had written for you, for each test case    | The evaluator's method, such as Checklist, Answer Relevancy, Faithfulness, Safety or Task Completion                                              |
| **How it's scored**              | An LLM scorer rates each reply from 0 to 10. A score of 7 or higher passes. | An evaluator, using a single judge model, a jury of models or one of your agents. A result below the evaluator's **Pass at or above** mark fails. |
| **Where you set it up**          | One agent, in **Agent Config › Regression Evals**                           | The account, in **Control Center › Evals**. An evaluator is assigned to agents and can serve more than one.                                       |
| **What you do with the results** | Read the failures, then apply **Suggested changes** to the agent            | Review flagged results in the **Review Queue**, and track quality in **Online Eval Runs**                                                         |
| **Effect on users**              | None. The test cases never reach a user.                                    | None. Scoring runs off the response path.                                                                                                         |

Use them together. Run Regression Evals before you release a change, to catch regressions against known cases. Use Evals after release, to catch drift on real traffic. When a reviewed miss in production reveals a gap, add it to a Regression Evals dataset so the next change is tested against it.

See [Evals in Control Center](https://docs.uptiq.ai/ai-workbench-qore/control-center/evals.md) and [Evals](https://docs.uptiq.ai/ai-workbench-qore/core-concepts/evals.md) for the production side.

</details>

| Capability           | Description                                                                                                                  | Limit                                             |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- |
| Dataset management   | Create, rename, delete datasets; import test cases from an uploaded document, or generate them from the agent's instructions | 100 per agent                                     |
| Test case management | Question, expected answer, optional document context and attachments                                                         | 50 per AI generation; up to 500 from one document |
| Evaluation runs      | Trigger, monitor progress in real-time, cancel, view results                                                                 | 100 per agent                                     |
| LLM scoring          | 0–10 scale with reasoning, 7+ = pass threshold                                                                               | —                                                 |
| Suggested changes    | AI-generated improvement recommendations with one-click apply                                                                | —                                                 |
| Document support     | Auto-generated DOCX/XLSX/PDF test documents with preview                                                                     | —                                                 |

## Open Regression Evals

Regression Evals belongs to one agent. Open the agent in Agent Builder and select **Agent Config**. The **Agent Config** window has a sidebar of settings in four groups, with no search box:

| Group                      | Settings                                                                                                                                 |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **Agent Setup**            | **Basic Details**, **Instructions**, **Agent Behaviour**, **Integrations**, **Skills & Capabilities**, **Triggers**, **Execution Plans** |
| **Credentials & access**   | **Secrets & Variables**, **Access & Security**, **Authentication**                                                                       |
| **Cognition & Control**    | **Guardrails**, **Regression Evals**                                                                                                     |
| **Reinforcement Learning** | **Learning Cases**, **FAQ**, **User Persona**                                                                                            |

Select **Regression Evals** under **Cognition & Control**, below **Guardrails**. The page has two tabs, **Datasets** and **Runs**. Inside a dataset or a run, a breadcrumb at the top, such as **Regression Evals › New dataset** or **Regression Evals › Evaluation run**, takes you back to the page.

## Workflow

{% tabs %}
{% tab title="Datasets" %}
Manage [datasets](#datasets) and [test cases](#test-cases).
{% endtab %}

{% tab title="Runs" %}
View [runs](#runs), [scoring](#scoring), and [suggested changes](#suggested-changes).
{% endtab %}
{% endtabs %}

{% stepper %}
{% step %}

### Build a dataset

Create a [dataset](#datasets) from an uploaded document, generate one with AI, or add test cases manually.
{% endstep %}

{% step %}

### Run the evaluation

Execute the dataset against the live agent in [Runs](#runs).
{% endstep %}

{% step %}

### Review and improve

Inspect [scores](#scoring), review failures, and apply [suggested changes](#suggested-changes).
{% endstep %}
{% endstepper %}

***

## Datasets

A dataset is a collection of test cases for evaluating an agent. Each agent can have up to **100 datasets**.

![](https://1326225582-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F0qmgQjJ5aArDTj2ACFHG%2Fuploads%2Fgit-blob-f9a93d4db65ffbdbf83e58481827d3f40af26ea1%2Fregression-datasets.png?alt=media)

### New dataset

Select **New dataset**, the green button at the top right of the **Datasets** tab. The **New dataset** window opens under the breadcrumb **Regression Evals › New dataset**. It reads `A dataset is a set of test cases your agent is evaluated against. Start empty, or fill it from AI or a document — you can add more test cases any way later.`

1. Enter a **Name** (required). The field suggests an example, such as `Refund policy questions`.
2. Under **How will you add test cases?**, choose one of three cards. **Write them yourself** is selected when the window opens, and carries a **New** label.
3. Fill in the fields the chosen card adds, if any.
4. Select the button at the foot of the window. **Cancel** closes the window without creating anything.

| Option                     | What it does                                                                       | Extra fields                                             | Button                  |
| -------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------- | ----------------------- |
| **Write them yourself**    | Start with an empty dataset and add each question and expected answer by hand.     | None                                                     | **Create dataset**      |
| **Generate with AI**       | Have test cases written for you from the agent's instructions.                     | **Number of test cases**, **What should they focus on?** | **Create and generate** |
| **Import from a document** | Upload a file that lists test cases, or one that describes the scenarios to cover. | **Document**                                             | **Create and import**   |

The window changes when you pick a card: the fields for that option appear below the cards, and the button at the foot takes the option's name.

### Create a dataset with AI or from a document

Both start in the **New dataset** window. The **Name** is required, up to 120 characters. Select **Create and generate** or **Create and import**, and the dataset opens at once with **Status** reading **Adding test cases**. A banner reads `New test cases are being written or read from your file. They appear here when ready — this can take a few minutes.` The test cases appear in the table when they are ready.

#### Generate with AI

Test cases are written from the agent's instructions.

| Field                          | Required | Description                                                                                                                                |
| ------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Number of test cases**       | Yes      | Up to 50 at a time. It starts at `10`. The hint under the field reads `Up to 50 at a time. They're written from the agent's instructions.` |
| **What should they focus on?** | No       | A free-text box that steers the test cases toward a theme. For example, `Edge cases around refunds and partial payments`.                  |

**Generation steps:**

1. The system reads the agent's system prompt (instructions)
2. Sends it to an LLM (Gemini 3 Flash) to generate test cases
3. For each test case, the LLM creates: question, expected answer, and optional document content
4. Document content is converted to actual files (DOCX, XLSX, or PDF) and uploaded to storage
5. Test case records are created with document references

#### Import from a document

Upload a document, and the test cases are read out of it for you. The document can list the test cases, or describe the scenarios to cover; from a description, the cases are derived from the rules and constraints it states.

The **Document** field is required. Drag a file onto the dashed area, or click the area to select one. A hint beneath it reads `A list of test cases, or a document describing the scenarios to cover · .pdf, .xlsx, .xls, .md · up to 10MB`. Select **Create and import** to create the dataset and start reading the file.

* **File types:** `.pdf`, `.xlsx`, `.xls`, `.md`
* **Size:** up to 10 MB
* **Test cases:** decided by the document, up to 500

#### Add test cases manually

Create manual test cases one at a time in an existing dataset.

On a dataset's page, select **Add test case**. The window reads `Write a question your agent should handle and the answer you expect back.` Its fields are **Question**, **Expected answer**, **Context (optional)** and **Attach documents (optional)**. Attached documents are the context the answer is checked against. They can be `.xlsx`, `.csv`, `.pdf`, `.jpg`, `.png`, `.docx`, `.doc`, `.webp`, `.heic`, `.heif`, `.txt` or `.md`, up to 10 more documents, and up to 50 MB per file. **Save and add another** keeps the window open for the next test case, and **Add test case** saves and closes it. The dataset's table then lists **Question**, **Expected answer** and **Documents**.

A test case's row menu has **Edit** and **Delete**. **Edit** opens **Edit test case**, which reads `Change the question or what a correct answer looks like.` It has the same fields as **Add test case**, with **Cancel** and **Save**.

Each manual test case can include:

* `question`
* `expected answer`
* optional documents

Use `POST /:agentId/eval/datasets/:datasetId/items` to create the test case.

Use **Test Case Search** to filter by question, expected answer, or document context.

<details>

<summary>How document format selection works</summary>

Document format selection is automatic.

| Format | Content Type             | Examples                      |
| ------ | ------------------------ | ----------------------------- |
| XLSX   | Tabular/spreadsheet data | Budgets, financial tables     |
| DOCX   | Narrative prose          | Memos, policies, contracts    |
| PDF    | Professional documents   | Invoices, resumes, statements |

</details>

### Dataset Views

The **Datasets** tab opens with four figures: **Datasets**, **Test cases**, **Eval runs**, and **Latest pass rate**. **Latest pass rate** is the share of test cases that passed in the most recent run.

**List view** has a filter button (the funnel icon) and a **Search** box above a table of the agent's datasets:

* **Name** — Dataset name
* **Status** — See the table below
* **Test cases** — How many test cases the dataset holds
* **Last run** — The pass rate of the dataset's last run, as a badge such as `33% passed`, or **Never run**
* **Created** — Creation timestamp, such as `Oct 26, 2026, 1:48 PM`
* **Actions** — The `⋮` menu at the end of the row has **View**, **Edit details** and **Delete**. **Edit details** opens **Edit dataset details**, where you change the **Name** and select **Save**. **Delete** asks `Delete dataset?` and says the dataset and its test cases will be removed and that this cannot be undone.

| Status                | Meaning                                                                                                                              |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| **Adding test cases** | The test cases are still being written or read from your file. They appear when ready.                                               |
| **Ready**             | The test cases are in place and the dataset can be run.                                                                              |
| **Failed**            | No test cases could be read from the document. Hover over the badge for the reason, then upload a revised document as a new dataset. |

An empty list reads **No results.**

**Details view** opens when you select **View**. The header is the dataset name, with **Add test case** and **Run evaluation**. Below it are **Status**, **Test cases**, **Last run** and **Created**, a **Search** box, and the table of [test cases](#test-cases).

***

### Test Cases

Each test case contains:

* **Question** — The input prompt sent to the agent
* **Expected answer** — The ground truth / ideal response
* **Context** (optional) — Context the answer is checked against, such as a description of an attached document (e.g., "Excel budget sheet")
* **Documents** (optional) — Files attached with **Attach documents**, shown as clickable chips

#### Document Preview

Clicking a document chip opens a preview modal:

| File Type                  | Viewer                                  |
| -------------------------- | --------------------------------------- |
| PDF                        | Native document viewer                  |
| Google Docs / Office files | Embedded Google or Office Online viewer |
| Excel                      | Office Online viewer                    |

{% hint style="info" %}
Toggle between "Show preview" and "Show context" if document context is available. Use the "Open file" button for external download.
{% endhint %}

***

## Runs

#### Starting a run

Open a dataset and select **Run evaluation**. The run starts at once and opens the **Evaluation run** page. It shows **Status**, **Progress** (for example `0 of 3`), **Passed**, **Failed**, **Avg score** and **Started**, and the line `Running test cases against your agent…`. **Stop evaluation** stops the run. Results appear in the table as each test case finishes.

A run gives no choice of method and no checklist. Checklists are a method on [evaluators](https://docs.uptiq.ai/ai-workbench-qore/control-center/evals/evaluators.md#checklist), which score live conversations in Online Eval.

#### Monitor runs

The **Runs** tab has the same filter button and **Search** box as the **Datasets** tab, above a table of runs with **Dataset**, **Status**, **Results**, **Avg Score**, **Started** and **Actions**:

* **Dataset** — The dataset the run used.
* **Status** — A badge, such as **Completed**. See [Run status](#run-status).
* **Results** — The count of test cases that passed and failed, for example `1 passed · 2 failed`.
* **Avg Score** — The average score out of 10, such as `5.0/10`.
* **Started** — When the run began, such as `Oct 26, 2026, 1:49 PM`.
* **Actions** — The `⋮` menu has one entry, **View**, which opens the **Evaluation run** page. A run whose dataset was deleted shows **Deleted** beside the dataset name and stays in the list.

#### The Evaluation run page

The page opens under the breadcrumb **Regression Evals › Evaluation run**. When a run completes, it has two tabs, **Results** and **Suggested changes**. Above them is a line of figures:

![](https://1326225582-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F0qmgQjJ5aArDTj2ACFHG%2Fuploads%2Fgit-blob-298584db3f049c2aab29efc4c86c78fa151c2581%2Fregression-run.png?alt=media)

| Figure        | Meaning                                                 |
| ------------- | ------------------------------------------------------- |
| **Status**    | The run's status, such as `Completed`.                  |
| **Progress**  | Test cases finished out of the total, such as `3 of 3`. |
| **Passed**    | How many test cases passed.                             |
| **Failed**    | How many test cases failed.                             |
| **Avg score** | The average score out of 10, such as `5.0/10`.          |
| **Started**   | When the run began.                                     |

Beneath the figures is a short written summary of the run. It names what the agent did well and where it fell short, for example that it handled summarization well but struggled to extract and compute from a PDF, and it points to the test cases involved.

The **Results** tab has a filter button and a **Search** box above the table. A footer such as `1-3 of 3` gives the number of rows. The table has these columns:

| Column              | What it holds                                                    |
| ------------------- | ---------------------------------------------------------------- |
| **Result**          | A **Pass** or **Fail** badge.                                    |
| **Question**        | The question sent to the agent. Long text is cut off in the row. |
| **Expected answer** | The answer the test case expects. Long text is cut off.          |
| **Actual answer**   | The agent's reply. Long text is cut off.                         |
| **Score**           | The score out of 10, such as `2/10` or `10/10`.                  |
| **Time**            | How long the test case took, such as `57s` or `1m 9s`.           |

A test case scores out of 10. In one run, scores of 7, 5 and 4 gave one pass and two fails, and the average read `5.3/10`. Select a row to open its [result](#scoring).

#### Run Execution

For each test case in the dataset:

1. Resolve document references to signed URLs
2. Trigger the agent via the Agent Executor (`/agents/{agentId}/trigger`)
3. Pass the question + documents to the agent
4. Extract the agent's actual answer from the response
5. Score the response using an LLM scorer (Gemini 3 Flash)
6. Record the result (score, reasoning, correctness, duration)
7. Update run progress counters

{% hint style="info" %}
Up to **3 test cases** execute concurrently during a run.
{% endhint %}

#### Run Status

<details>

<summary>Status lifecycle</summary>

{% code collapsedlinecount="10" %}

```
Pending → Running → Completed
                 → Failed
                 → Stopped (user-stopped)
```

{% endcode %}

</details>

| Status        | Indicator        | Details Shown                               | Available Actions                                     |
| ------------- | ---------------- | ------------------------------------------- | ----------------------------------------------------- |
| **Pending**   | Warning badge    | Waiting to start                            | —                                                     |
| **Running**   | Animated spinner | Progress bar (completedItems / totalItems)  | Stop                                                  |
| **Completed** | Green badge      | Pass/fail counts, [average score](#scoring) | View results, Apply [suggestions](#suggested-changes) |
| **Failed**    | Red badge        | Error message                               | —                                                     |
| **Stopped**   | Red badge        | Partial [results](#scoring)                 | View partial [results](#scoring)                      |

{% hint style="info" %}
A **Stop** button appears during in-progress runs. Stopping preserves already-completed results. The UI polls every **3 seconds** while a run is in progress, stopping automatically when the run completes.
{% endhint %}

***

## Scoring

**Score Scale:** Each test case response is scored by an LLM (Gemini 3 Flash):

| Score    | Verdict           | Description                                          | Result |
| -------- | ----------------- | ---------------------------------------------------- | ------ |
| **0–3**  | Incorrect         | Response is wrong or irrelevant                      | Fail   |
| **4–6**  | Partially correct | Some relevant content but incomplete or inaccurate   | Fail   |
| **7–8**  | Correct           | Meets the expected answer criteria                   | Pass   |
| **9–10** | Comprehensive     | Exceeds expectations with additional relevant detail | Pass   |

{% hint style="warning" %}
**Pass threshold:** A score of 7 or above marks the test case as passed.
{% endhint %}

**Test case result** — select a row in the results table to open it. It shows:

* **Result** — Pass (green), Fail (red), or Error (red)
* **Score** — Such as `7/10`
* **Time taken** — Execution time
* **Question** — Original input
* **Expected answer** — Ground truth
* **Actual answer** — Agent's response (or error message)
* **Scorer reasoning** — LLM's explanation of the score

Results appear progressively as each test case completes.

{% hint style="success" %}
Completed runs display **pass count** (green), **fail count** (red), **average score** (0–10), and an LLM-generated **summary** of evaluation results.
{% endhint %}

<details>

<summary>What a failing run usually means</summary>

A failing run does not always mean the agent is broken.

Common causes include:

* incomplete instructions
* missing skills or knowledge
* answers that are directionally correct but below the pass threshold

</details>

***

## Suggested Changes

{% hint style="info" %}
Suggestions are generated automatically when a run completes with failures or an average score below 7.
{% endhint %}

They are listed on the run's **Suggested changes** tab, below the same line of run figures and written summary as the **Results** tab. **Apply to Agent**, a green button at the top right of the page, applies them. Each suggestion is a row with three columns:

* **Change** — Short description (max 200 characters), such as `Improve PDF data extraction capabilities`
* **Area** — A label for what the change touches: Instructions, Skills, Knowledge, Guardrails, or Other
* **Why** — Detailed recommendation (max 2,000 characters). For example, a **Skills** change to improve PDF data extraction explains that the agent should recognise itemized entries and add up the amounts to reach the totals the task asks for. An **Instructions** change to follow the task's format explains that the extracted data must include every requested field.

Suggestions can modify: agent instructions, skills, knowledge, and guardrails.

#### Applying Suggestions

{% stepper %}
{% step %}

### Open suggestions

Select **Apply to Agent** at the top of the **Suggested changes** tab of a completed run.
{% endstep %}

{% step %}

### Review selections

The **Apply evaluation suggestions** window reads `Select which suggestions to apply and edit them before sending to the builder agent. Progress appears in the builder chat panel on the left.` It shows a count such as `2 of 2 selected`, with **Select all** and **Clear all**.

Edit the **Title**, **Category**, or **Description** of a suggestion as needed.
{% endstep %}

{% step %}

### Apply changes

Select the suggestions to apply, then select **Apply** with the count, such as **Apply 2 suggestions**.

Toast: "Builder agent is applying your changes — watch the chat panel on the left"
{% endstep %}

{% step %}

### Continue in chat

The agent config modal closes and the Builder Agent executes the changes in the chat panel.
{% endstep %}
{% endstepper %}

### Related sections

* [Monitoring](https://docs.uptiq.ai/ai-workbench-qore/console/agent-builder/monitor.md) — Watch runtime behavior after the agent is live.
* [Online Evals in Control Center](https://docs.uptiq.ai/ai-workbench-qore/control-center/evals.md) — Score live conversations in production.
* [Reinforcement Learnings](https://docs.uptiq.ai/ai-workbench-qore/console/agent-builder/monitor/reinforcement-learnings.md) — Review and approve learning cases from real usage.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.uptiq.ai/ai-workbench-qore/console/agent-builder/test/regression-evals.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
