Regression Evals
Test agent responses against scored datasets with automated LLM grading.
Regression Evals is an evaluation framework for systematically testing agent quality on the Uptiq Platform. It allows teams to create datasets of test cases — read from an uploaded document, generated by an LLM, or added manually — run those test cases against a live agent, score each response on a 0–10 scale using an LLM scorer, and receive AI-generated suggestions for improving the agent.
Dataset management
Create, rename, delete datasets; import test cases from an uploaded document, or generate them from the agent's instructions
100 per agent
Test case management
Question, expected answer, optional document context and attachments
50 per AI generation; up to 500 from one document
Evaluation runs
Trigger, monitor progress in real-time, cancel, view results
100 per agent
LLM scoring
0–10 scale with reasoning, 7+ = pass threshold
—
Suggested changes
AI-generated improvement recommendations with one-click apply
—
Document support
Auto-generated DOCX/XLSX/PDF test documents with preview
—
Open Regression Evals
Regression Evals belongs to one agent. Open the agent in Agent Builder and select Agent Config. The Agent Config window has a sidebar of settings in four groups, with no search box:
Agent Setup
Basic Details, Instructions, Agent Behaviour, Integrations, Skills & Capabilities, Triggers, Execution Plans
Credentials & access
Secrets & Variables, Access & Security, Authentication
Cognition & Control
Guardrails, Regression Evals
Reinforcement Learning
Learning Cases, FAQ, User Persona
Select Regression Evals under Cognition & Control, below Guardrails. The page has two tabs, Datasets and Runs. Inside a dataset or a run, a breadcrumb at the top, such as Regression Evals › New dataset or Regression Evals › Evaluation run, takes you back to the page.
Workflow
Manage datasets and test cases.
View runs, scoring, and suggested changes.
Build a dataset
Create a dataset from an uploaded document, generate one with AI, or add test cases manually.
Run the evaluation
Execute the dataset against the live agent in Runs.
Review and improve
Inspect scores, review failures, and apply suggested changes.
Datasets
A dataset is a collection of test cases for evaluating an agent. Each agent can have up to 100 datasets.

New dataset
Select New dataset, the green button at the top right of the Datasets tab. The New dataset window opens under the breadcrumb Regression Evals › New dataset. It reads A dataset is a set of test cases your agent is evaluated against. Start empty, or fill it from AI or a document — you can add more test cases any way later.
Enter a Name (required). The field suggests an example, such as
Refund policy questions.Under How will you add test cases?, choose one of three cards. Write them yourself is selected when the window opens, and carries a New label.
Fill in the fields the chosen card adds, if any.
Select the button at the foot of the window. Cancel closes the window without creating anything.
Write them yourself
Start with an empty dataset and add each question and expected answer by hand.
None
Create dataset
Generate with AI
Have test cases written for you from the agent's instructions.
Number of test cases, What should they focus on?
Create and generate
Import from a document
Upload a file that lists test cases, or one that describes the scenarios to cover.
Document
Create and import
The window changes when you pick a card: the fields for that option appear below the cards, and the button at the foot takes the option's name.
Create a dataset with AI or from a document
Both start in the New dataset window. The Name is required, up to 120 characters. Select Create and generate or Create and import, and the dataset opens at once with Status reading Adding test cases. A banner reads New test cases are being written or read from your file. They appear here when ready — this can take a few minutes. The test cases appear in the table when they are ready.
Generate with AI
Test cases are written from the agent's instructions.
Number of test cases
Yes
Up to 50 at a time. It starts at 10. The hint under the field reads Up to 50 at a time. They're written from the agent's instructions.
What should they focus on?
No
A free-text box that steers the test cases toward a theme. For example, Edge cases around refunds and partial payments.
Generation steps:
The system reads the agent's system prompt (instructions)
Sends it to an LLM (Gemini 3 Flash) to generate test cases
For each test case, the LLM creates: question, expected answer, and optional document content
Document content is converted to actual files (DOCX, XLSX, or PDF) and uploaded to storage
Test case records are created with document references
Import from a document
Upload a document, and the test cases are read out of it for you. The document can list the test cases, or describe the scenarios to cover; from a description, the cases are derived from the rules and constraints it states.
The Document field is required. Drag a file onto the dashed area, or click the area to select one. A hint beneath it reads A list of test cases, or a document describing the scenarios to cover · .pdf, .xlsx, .xls, .md · up to 10MB. Select Create and import to create the dataset and start reading the file.
File types:
.pdf,.xlsx,.xls,.mdSize: up to 10 MB
Test cases: decided by the document, up to 500
Add test cases manually
Create manual test cases one at a time in an existing dataset.
On a dataset's page, select Add test case. The window reads Write a question your agent should handle and the answer you expect back. Its fields are Question, Expected answer, Context (optional) and Attach documents (optional). Attached documents are the context the answer is checked against. They can be .xlsx, .csv, .pdf, .jpg, .png, .docx, .doc, .webp, .heic, .heif, .txt or .md, up to 10 more documents, and up to 50 MB per file. Save and add another keeps the window open for the next test case, and Add test case saves and closes it. The dataset's table then lists Question, Expected answer and Documents.
A test case's row menu has Edit and Delete. Edit opens Edit test case, which reads Change the question or what a correct answer looks like. It has the same fields as Add test case, with Cancel and Save.
Each manual test case can include:
questionexpected answeroptional documents
Use POST /:agentId/eval/datasets/:datasetId/items to create the test case.
Use Test Case Search to filter by question, expected answer, or document context.
Dataset Views
The Datasets tab opens with four figures: Datasets, Test cases, Eval runs, and Latest pass rate. Latest pass rate is the share of test cases that passed in the most recent run.
List view has a filter button (the funnel icon) and a Search box above a table of the agent's datasets:
Name — Dataset name
Status — See the table below
Test cases — How many test cases the dataset holds
Last run — The pass rate of the dataset's last run, as a badge such as
33% passed, or Never runCreated — Creation timestamp, such as
Oct 26, 2026, 1:48 PMActions — The
⋮menu at the end of the row has View, Edit details and Delete. Edit details opens Edit dataset details, where you change the Name and select Save. Delete asksDelete dataset?and says the dataset and its test cases will be removed and that this cannot be undone.
Adding test cases
The test cases are still being written or read from your file. They appear when ready.
Ready
The test cases are in place and the dataset can be run.
Failed
No test cases could be read from the document. Hover over the badge for the reason, then upload a revised document as a new dataset.
An empty list reads No results.
Details view opens when you select View. The header is the dataset name, with Add test case and Run evaluation. Below it are Status, Test cases, Last run and Created, a Search box, and the table of test cases.
Test Cases
Each test case contains:
Question — The input prompt sent to the agent
Expected answer — The ground truth / ideal response
Context (optional) — Context the answer is checked against, such as a description of an attached document (e.g., "Excel budget sheet")
Documents (optional) — Files attached with Attach documents, shown as clickable chips
Document Preview
Clicking a document chip opens a preview modal:
Native document viewer
Google Docs / Office files
Embedded Google or Office Online viewer
Excel
Office Online viewer
Runs
Starting a run
Open a dataset and select Run evaluation. The run starts at once and opens the Evaluation run page. It shows Status, Progress (for example 0 of 3), Passed, Failed, Avg score and Started, and the line Running test cases against your agent…. Stop evaluation stops the run. Results appear in the table as each test case finishes.
A run gives no choice of method and no checklist. Checklists are a method on evaluators, which score live conversations in Online Eval.
Monitor runs
The Runs tab has the same filter button and Search box as the Datasets tab, above a table of runs with Dataset, Status, Results, Avg Score, Started and Actions:
Dataset — The dataset the run used.
Status — A badge, such as Completed. See Run status.
Results — The count of test cases that passed and failed, for example
1 passed · 2 failed.Avg Score — The average score out of 10, such as
5.0/10.Started — When the run began, such as
Oct 26, 2026, 1:49 PM.Actions — The
⋮menu has one entry, View, which opens the Evaluation run page. A run whose dataset was deleted shows Deleted beside the dataset name and stays in the list.
The Evaluation run page
The page opens under the breadcrumb Regression Evals › Evaluation run. When a run completes, it has two tabs, Results and Suggested changes. Above them is a line of figures:

Status
The run's status, such as Completed.
Progress
Test cases finished out of the total, such as 3 of 3.
Passed
How many test cases passed.
Failed
How many test cases failed.
Avg score
The average score out of 10, such as 5.0/10.
Started
When the run began.
Beneath the figures is a short written summary of the run. It names what the agent did well and where it fell short, for example that it handled summarization well but struggled to extract and compute from a PDF, and it points to the test cases involved.
The Results tab has a filter button and a Search box above the table. A footer such as 1-3 of 3 gives the number of rows. The table has these columns:
Result
A Pass or Fail badge.
Question
The question sent to the agent. Long text is cut off in the row.
Expected answer
The answer the test case expects. Long text is cut off.
Actual answer
The agent's reply. Long text is cut off.
Score
The score out of 10, such as 2/10 or 10/10.
Time
How long the test case took, such as 57s or 1m 9s.
A test case scores out of 10. In one run, scores of 7, 5 and 4 gave one pass and two fails, and the average read 5.3/10. Select a row to open its result.
Run Execution
For each test case in the dataset:
Resolve document references to signed URLs
Trigger the agent via the Agent Executor (
/agents/{agentId}/trigger)Pass the question + documents to the agent
Extract the agent's actual answer from the response
Score the response using an LLM scorer (Gemini 3 Flash)
Record the result (score, reasoning, correctness, duration)
Update run progress counters
Run Status
Pending
Warning badge
Waiting to start
—
Running
Animated spinner
Progress bar (completedItems / totalItems)
Stop
Failed
Red badge
Error message
—
Scoring
Score Scale: Each test case response is scored by an LLM (Gemini 3 Flash):
0–3
Incorrect
Response is wrong or irrelevant
Fail
4–6
Partially correct
Some relevant content but incomplete or inaccurate
Fail
7–8
Correct
Meets the expected answer criteria
Pass
9–10
Comprehensive
Exceeds expectations with additional relevant detail
Pass
Test case result — select a row in the results table to open it. It shows:
Result — Pass (green), Fail (red), or Error (red)
Score — Such as
7/10Time taken — Execution time
Question — Original input
Expected answer — Ground truth
Actual answer — Agent's response (or error message)
Scorer reasoning — LLM's explanation of the score
Results appear progressively as each test case completes.
Suggested Changes
They are listed on the run's Suggested changes tab, below the same line of run figures and written summary as the Results tab. Apply to Agent, a green button at the top right of the page, applies them. Each suggestion is a row with three columns:
Change — Short description (max 200 characters), such as
Improve PDF data extraction capabilitiesArea — A label for what the change touches: Instructions, Skills, Knowledge, Guardrails, or Other
Why — Detailed recommendation (max 2,000 characters). For example, a Skills change to improve PDF data extraction explains that the agent should recognise itemized entries and add up the amounts to reach the totals the task asks for. An Instructions change to follow the task's format explains that the extracted data must include every requested field.
Suggestions can modify: agent instructions, skills, knowledge, and guardrails.
Applying Suggestions
Review selections
The Apply evaluation suggestions window reads Select which suggestions to apply and edit them before sending to the builder agent. Progress appears in the builder chat panel on the left. It shows a count such as 2 of 2 selected, with Select all and Clear all.
Edit the Title, Category, or Description of a suggestion as needed.
Related sections
Monitoring — Watch runtime behavior after the agent is live.
Online Evals in Control Center — Score live conversations in production.
Reinforcement Learnings — Review and approve learning cases from real usage.
Last updated
Was this helpful?

