Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.
Cognition & Control

Regression Evals

Test agent responses against scored datasets with automated LLM grading.

Regression Evals is an evaluation framework for systematically testing agent quality on the Uptiq Platform. It allows teams to create datasets of test cases — read from an uploaded document, generated by an LLM, or added manually — run those test cases against a live agent, score each response on a 0–10 scale using an LLM scorer, and receive AI-generated suggestions for improving the agent.

Use Regression Evals before release, Online Evals in Control Center after release, and Reinforcement Learnings to carry reviewed feedback forward.

Regression Evals or Evals? The difference

Qore has two evaluation features, and their names are close. Regression Evals is the one on this page. Evals is a separate section of Control Center. They answer different questions.

Regression Evals

Evals (Control Center)

Question it answers

Does the agent still give the right answers after I change it?

How good are the answers the agent gives to real users?

When it runs

Before release, when you choose to run it

After release, in the background on live conversations

What is scored

The agent's replies to your test cases

Real executions, which the agent ran for real users

What you compare against

An expected answer you wrote, or had written for you, for each test case

The evaluator's method, such as Checklist, Answer Relevancy, Faithfulness, Safety or Task Completion

How it's scored

An LLM scorer rates each reply from 0 to 10. A score of 7 or higher passes.

An evaluator, using a single judge model, a jury of models or one of your agents. A result below the evaluator's Pass at or above mark fails.

Where you set it up

One agent, in Agent Config › Regression Evals

The account, in Control Center › Evals. An evaluator is assigned to agents and can serve more than one.

What you do with the results

Read the failures, then apply Suggested changes to the agent

Review flagged results in the Review Queue, and track quality in Online Eval Runs

Effect on users

None. The test cases never reach a user.

None. Scoring runs off the response path.

Use them together. Run Regression Evals before you release a change, to catch regressions against known cases. Use Evals after release, to catch drift on real traffic. When a reviewed miss in production reveals a gap, add it to a Regression Evals dataset so the next change is tested against it.

See Evals in Control Center and Evals for the production side.

Capability
Description
Limit

Dataset management

Create, rename, delete datasets; import test cases from an uploaded document, or generate them from the agent's instructions

100 per agent

Test case management

Question, expected answer, optional document context and attachments

50 per AI generation; up to 500 from one document

Evaluation runs

Trigger, monitor progress in real-time, cancel, view results

100 per agent

LLM scoring

0–10 scale with reasoning, 7+ = pass threshold

—

Suggested changes

AI-generated improvement recommendations with one-click apply

—

Document support

Auto-generated DOCX/XLSX/PDF test documents with preview

—

Open Regression Evals

Regression Evals belongs to one agent. Open the agent in Agent Builder and select Agent Config. The Agent Config window has a sidebar of settings in four groups, with no search box:

Group
Settings

Agent Setup

Basic Details, Instructions, Agent Behaviour, Integrations, Skills & Capabilities, Triggers, Execution Plans

Credentials & access

Secrets & Variables, Access & Security, Authentication

Cognition & Control

Guardrails, Regression Evals

Reinforcement Learning

Learning Cases, FAQ, User Persona

Select Regression Evals under Cognition & Control, below Guardrails. The page has two tabs, Datasets and Runs. Inside a dataset or a run, a breadcrumb at the top, such as Regression Evals › New dataset or Regression Evals › Evaluation run, takes you back to the page.

Workflow

Manage datasets and test cases.

1

Build a dataset

Create a dataset from an uploaded document, generate one with AI, or add test cases manually.

2

Run the evaluation

Execute the dataset against the live agent in Runs.

3

Review and improve

Inspect scores, review failures, and apply suggested changes.


Datasets

A dataset is a collection of test cases for evaluating an agent. Each agent can have up to 100 datasets.

New dataset

Select New dataset, the green button at the top right of the Datasets tab. The New dataset window opens under the breadcrumb Regression Evals › New dataset. It reads A dataset is a set of test cases your agent is evaluated against. Start empty, or fill it from AI or a document — you can add more test cases any way later.

  1. Enter a Name (required). The field suggests an example, such as Refund policy questions.

  2. Under How will you add test cases?, choose one of three cards. Write them yourself is selected when the window opens, and carries a New label.

  3. Fill in the fields the chosen card adds, if any.

  4. Select the button at the foot of the window. Cancel closes the window without creating anything.

Option
What it does
Extra fields
Button

Write them yourself

Start with an empty dataset and add each question and expected answer by hand.

None

Create dataset

Generate with AI

Have test cases written for you from the agent's instructions.

Number of test cases, What should they focus on?

Create and generate

Import from a document

Upload a file that lists test cases, or one that describes the scenarios to cover.

Document

Create and import

The window changes when you pick a card: the fields for that option appear below the cards, and the button at the foot takes the option's name.

Create a dataset with AI or from a document

Both start in the New dataset window. The Name is required, up to 120 characters. Select Create and generate or Create and import, and the dataset opens at once with Status reading Adding test cases. A banner reads New test cases are being written or read from your file. They appear here when ready — this can take a few minutes. The test cases appear in the table when they are ready.

Generate with AI

Test cases are written from the agent's instructions.

Field
Required
Description

Number of test cases

Yes

Up to 50 at a time. It starts at 10. The hint under the field reads Up to 50 at a time. They're written from the agent's instructions.

What should they focus on?

No

A free-text box that steers the test cases toward a theme. For example, Edge cases around refunds and partial payments.

Generation steps:

  1. The system reads the agent's system prompt (instructions)

  2. Sends it to an LLM (Gemini 3 Flash) to generate test cases

  3. For each test case, the LLM creates: question, expected answer, and optional document content

  4. Document content is converted to actual files (DOCX, XLSX, or PDF) and uploaded to storage

  5. Test case records are created with document references

Import from a document

Upload a document, and the test cases are read out of it for you. The document can list the test cases, or describe the scenarios to cover; from a description, the cases are derived from the rules and constraints it states.

The Document field is required. Drag a file onto the dashed area, or click the area to select one. A hint beneath it reads A list of test cases, or a document describing the scenarios to cover · .pdf, .xlsx, .xls, .md · up to 10MB. Select Create and import to create the dataset and start reading the file.

  • File types: .pdf, .xlsx, .xls, .md

  • Size: up to 10 MB

  • Test cases: decided by the document, up to 500

Add test cases manually

Create manual test cases one at a time in an existing dataset.

On a dataset's page, select Add test case. The window reads Write a question your agent should handle and the answer you expect back. Its fields are Question, Expected answer, Context (optional) and Attach documents (optional). Attached documents are the context the answer is checked against. They can be .xlsx, .csv, .pdf, .jpg, .png, .docx, .doc, .webp, .heic, .heif, .txt or .md, up to 10 more documents, and up to 50 MB per file. Save and add another keeps the window open for the next test case, and Add test case saves and closes it. The dataset's table then lists Question, Expected answer and Documents.

A test case's row menu has Edit and Delete. Edit opens Edit test case, which reads Change the question or what a correct answer looks like. It has the same fields as Add test case, with Cancel and Save.

Each manual test case can include:

  • question

  • expected answer

  • optional documents

Use POST /:agentId/eval/datasets/:datasetId/items to create the test case.

Use Test Case Search to filter by question, expected answer, or document context.

How document format selection works

Document format selection is automatic.

Format
Content Type
Examples

XLSX

Tabular/spreadsheet data

Budgets, financial tables

DOCX

Narrative prose

Memos, policies, contracts

PDF

Professional documents

Invoices, resumes, statements

Dataset Views

The Datasets tab opens with four figures: Datasets, Test cases, Eval runs, and Latest pass rate. Latest pass rate is the share of test cases that passed in the most recent run.

List view has a filter button (the funnel icon) and a Search box above a table of the agent's datasets:

  • Name — Dataset name

  • Status — See the table below

  • Test cases — How many test cases the dataset holds

  • Last run — The pass rate of the dataset's last run, as a badge such as 33% passed, or Never run

  • Created — Creation timestamp, such as Oct 26, 2026, 1:48 PM

  • Actions — The ⋮ menu at the end of the row has View, Edit details and Delete. Edit details opens Edit dataset details, where you change the Name and select Save. Delete asks Delete dataset? and says the dataset and its test cases will be removed and that this cannot be undone.

Status
Meaning

Adding test cases

The test cases are still being written or read from your file. They appear when ready.

Ready

The test cases are in place and the dataset can be run.

Failed

No test cases could be read from the document. Hover over the badge for the reason, then upload a revised document as a new dataset.

An empty list reads No results.

Details view opens when you select View. The header is the dataset name, with Add test case and Run evaluation. Below it are Status, Test cases, Last run and Created, a Search box, and the table of test cases.


Test Cases

Each test case contains:

  • Question — The input prompt sent to the agent

  • Expected answer — The ground truth / ideal response

  • Context (optional) — Context the answer is checked against, such as a description of an attached document (e.g., "Excel budget sheet")

  • Documents (optional) — Files attached with Attach documents, shown as clickable chips

Document Preview

Clicking a document chip opens a preview modal:

File Type
Viewer

PDF

Native document viewer

Google Docs / Office files

Embedded Google or Office Online viewer

Excel

Office Online viewer

Toggle between "Show preview" and "Show context" if document context is available. Use the "Open file" button for external download.


Runs

Starting a run

Open a dataset and select Run evaluation. The run starts at once and opens the Evaluation run page. It shows Status, Progress (for example 0 of 3), Passed, Failed, Avg score and Started, and the line Running test cases against your agent…. Stop evaluation stops the run. Results appear in the table as each test case finishes.

A run gives no choice of method and no checklist. Checklists are a method on evaluators, which score live conversations in Online Eval.

Monitor runs

The Runs tab has the same filter button and Search box as the Datasets tab, above a table of runs with Dataset, Status, Results, Avg Score, Started and Actions:

  • Dataset — The dataset the run used.

  • Status — A badge, such as Completed. See Run status.

  • Results — The count of test cases that passed and failed, for example 1 passed · 2 failed.

  • Avg Score — The average score out of 10, such as 5.0/10.

  • Started — When the run began, such as Oct 26, 2026, 1:49 PM.

  • Actions — The ⋮ menu has one entry, View, which opens the Evaluation run page. A run whose dataset was deleted shows Deleted beside the dataset name and stays in the list.

The Evaluation run page

The page opens under the breadcrumb Regression Evals › Evaluation run. When a run completes, it has two tabs, Results and Suggested changes. Above them is a line of figures:

Figure
Meaning

Status

The run's status, such as Completed.

Progress

Test cases finished out of the total, such as 3 of 3.

Passed

How many test cases passed.

Failed

How many test cases failed.

Avg score

The average score out of 10, such as 5.0/10.

Started

When the run began.

Beneath the figures is a short written summary of the run. It names what the agent did well and where it fell short, for example that it handled summarization well but struggled to extract and compute from a PDF, and it points to the test cases involved.

The Results tab has a filter button and a Search box above the table. A footer such as 1-3 of 3 gives the number of rows. The table has these columns:

Column
What it holds

Result

A Pass or Fail badge.

Question

The question sent to the agent. Long text is cut off in the row.

Expected answer

The answer the test case expects. Long text is cut off.

Actual answer

The agent's reply. Long text is cut off.

Score

The score out of 10, such as 2/10 or 10/10.

Time

How long the test case took, such as 57s or 1m 9s.

A test case scores out of 10. In one run, scores of 7, 5 and 4 gave one pass and two fails, and the average read 5.3/10. Select a row to open its result.

Run Execution

For each test case in the dataset:

  1. Resolve document references to signed URLs

  2. Trigger the agent via the Agent Executor (/agents/{agentId}/trigger)

  3. Pass the question + documents to the agent

  4. Extract the agent's actual answer from the response

  5. Score the response using an LLM scorer (Gemini 3 Flash)

  6. Record the result (score, reasoning, correctness, duration)

  7. Update run progress counters

Up to 3 test cases execute concurrently during a run.

Run Status

Status lifecycle
Status
Indicator
Details Shown
Available Actions

Pending

Warning badge

Waiting to start

—

Running

Animated spinner

Progress bar (completedItems / totalItems)

Stop

Completed

Green badge

Pass/fail counts, average score

View results, Apply suggestions

Failed

Red badge

Error message

—

Stopped

Red badge

Partial results

View partial results

A Stop button appears during in-progress runs. Stopping preserves already-completed results. The UI polls every 3 seconds while a run is in progress, stopping automatically when the run completes.


Scoring

Score Scale: Each test case response is scored by an LLM (Gemini 3 Flash):

Score
Verdict
Description
Result

0–3

Incorrect

Response is wrong or irrelevant

Fail

4–6

Partially correct

Some relevant content but incomplete or inaccurate

Fail

7–8

Correct

Meets the expected answer criteria

Pass

9–10

Comprehensive

Exceeds expectations with additional relevant detail

Pass

Test case result — select a row in the results table to open it. It shows:

  • Result — Pass (green), Fail (red), or Error (red)

  • Score — Such as 7/10

  • Time taken — Execution time

  • Question — Original input

  • Expected answer — Ground truth

  • Actual answer — Agent's response (or error message)

  • Scorer reasoning — LLM's explanation of the score

Results appear progressively as each test case completes.

What a failing run usually means

A failing run does not always mean the agent is broken.

Common causes include:

  • incomplete instructions

  • missing skills or knowledge

  • answers that are directionally correct but below the pass threshold


Suggested Changes

Suggestions are generated automatically when a run completes with failures or an average score below 7.

They are listed on the run's Suggested changes tab, below the same line of run figures and written summary as the Results tab. Apply to Agent, a green button at the top right of the page, applies them. Each suggestion is a row with three columns:

  • Change — Short description (max 200 characters), such as Improve PDF data extraction capabilities

  • Area — A label for what the change touches: Instructions, Skills, Knowledge, Guardrails, or Other

  • Why — Detailed recommendation (max 2,000 characters). For example, a Skills change to improve PDF data extraction explains that the agent should recognise itemized entries and add up the amounts to reach the totals the task asks for. An Instructions change to follow the task's format explains that the extracted data must include every requested field.

Suggestions can modify: agent instructions, skills, knowledge, and guardrails.

Applying Suggestions

1

Open suggestions

Select Apply to Agent at the top of the Suggested changes tab of a completed run.

2

Review selections

The Apply evaluation suggestions window reads Select which suggestions to apply and edit them before sending to the builder agent. Progress appears in the builder chat panel on the left. It shows a count such as 2 of 2 selected, with Select all and Clear all.

Edit the Title, Category, or Description of a suggestion as needed.

3

Apply changes

Select the suggestions to apply, then select Apply with the count, such as Apply 2 suggestions.

Toast: "Builder agent is applying your changes — watch the chat panel on the left"

4

Continue in chat

The agent config modal closes and the Builder Agent executes the changes in the chat panel.

Last updated

Was this helpful?