Evals
Test agent responses against scored datasets with automated LLM grading.
Evals is an evaluation framework for systematically testing agent quality on the Uptiq Platform. It allows teams to create datasets of test cases — manually or auto-generated by an LLM — run those test cases against a live agent, score each response on a 0–10 scale using an LLM scorer, and receive AI-generated suggestions for improving the agent.
Use Evals before release, Online-Eval after release, and Reinforcement Learnings to carry reviewed feedback forward.
Dataset management
Create, rename, delete datasets; auto-generate test cases from agent config via LLM
100 per agent
Test case management
Question, expected answer, optional document context and attachments
50 per generation
Evaluation runs
Trigger, monitor progress in real-time, cancel, view results
100 per agent
LLM scoring
0–10 scale with reasoning, 7+ = pass threshold
—
Suggested changes
AI-generated improvement recommendations with one-click apply
—
Document support
Auto-generated DOCX/XLSX/PDF test documents with preview
—
Evals is accessed from the Cognition & Control section in the Agent Builder's Manage Agent sidebar.
Workflow
Build a dataset
Create a dataset manually or generate one with AI.
Run the evaluation
Execute the dataset against the live agent in Runs.
Review and improve
Inspect scores, review failures, and apply suggested changes.
Datasets
A dataset is a collection of test cases for evaluating an agent. Each agent can have up to 100 datasets.
Generate Dataset
Users click "Generate Dataset" to auto-generate test cases.
Dataset Name
Yes
Defaults to "Eval Dataset {date}"
Number of Test Cases
Yes
1–50, defaults to 10
Additional Instructions
No
e.g., "Focus on edge cases related to pricing"
Generation steps:
The system reads the agent's system prompt (instructions)
Sends it to an LLM (Gemini 3 Flash) to generate test cases
For each test case, the LLM creates: question, expected answer, and optional document content
Document content is converted to actual files (DOCX, XLSX, or PDF) and uploaded to storage
Test case records are created with document references
Create manual test cases one at a time in an existing dataset.
Each manual test case can include:
questionexpected answeroptional documents
Use POST /:agentId/eval/datasets/:datasetId/items to create the test case.
Use Test Case Search to filter by question, expected answer, or document context.
How document format selection works
Document format selection is automatic.
XLSX
Tabular/spreadsheet data
Budgets, financial tables
DOCX
Narrative prose
Memos, policies, contracts
Professional documents
Invoices, resumes, statements
Dataset Views
List view displays all datasets for the agent:
Name — Dataset name
Created — Creation timestamp
Actions — Rename, Delete
Empty state: "No evaluation datasets yet. Generate one to get started."
Details view shows the dataset name as header with a "Run Evaluation" button (disabled if no test cases), plus the list of all test cases.
Test Cases
Each test case contains:
Question — The input prompt sent to the agent
Expected Answer — The ground truth / ideal response
Document Context (optional) — Description of attached documents (e.g., "Excel budget sheet")
Documents (optional) — Attached files (DOCX, XLSX, PDF) shown as clickable chips
Document Preview
Clicking a document chip opens a preview modal:
Native document viewer
Google Docs / Office files
Embedded Google or Office Online viewer
Excel
Office Online viewer
Toggle between "Show preview" and "Show context" if document context is available. Use the "Open file" button for external download.
Runs
Starting a Run
Navigate to a dataset with at least one test case
Click "Run Evaluation"
The system creates an eval run with status
PendingBackground processing starts
Success toast: "Evaluation run started"
Use the Runs tab to track:
current status
progress
scores
suggestions
Run Execution
For each test case in the dataset:
Resolve document references to signed URLs
Trigger the agent via the Agent Executor (
/agents/{agentId}/trigger)Pass the question + documents to the agent
Extract the agent's actual answer from the response
Score the response using an LLM scorer (Gemini 3 Flash)
Record the result (score, reasoning, correctness, duration)
Update run progress counters
Up to 3 test cases execute concurrently during a run.
Run Status
Pending
Warning badge
Waiting to start
—
Running
Animated spinner
Progress bar (completedItems / totalItems)
Stop
Failed
Red badge
Error message
—
A Stop button appears during in-progress runs. Stopping preserves already-completed results. The UI polls every 3 seconds while a run is in progress, stopping automatically when the run completes.
Scoring
Score Scale: Each test case response is scored by an LLM (Gemini 3 Flash):
0–3
Incorrect
Response is wrong or irrelevant
Fail
4–6
Partially correct
Some relevant content but incomplete or inaccurate
Fail
7–8
Correct
Meets the expected answer criteria
Pass
9–10
Comprehensive
Exceeds expectations with additional relevant detail
Pass
Pass threshold: A score of 7 or above marks the test case as passed.
Result Card — each completed test case displays:
Status Badge — Pass (green), Fail (red), or Error (red)
Score — "Score: X/10"
Duration — Execution time
Question — Original input
Expected Answer — Ground truth (muted, scrollable)
Actual Answer — Agent's response (or error message)
Scorer Reasoning — LLM's explanation of the score
Results appear progressively as each test case completes.
Completed runs display pass count (green), fail count (red), average score (0–10), and an LLM-generated summary of evaluation results.
What a failing run usually means
A failing run does not always mean the agent is broken.
Common causes include:
incomplete instructions
missing skills or knowledge
answers that are directionally correct but below the pass threshold
Suggested Changes
Suggestions are generated automatically when a run completes with failures or an average score below 7.
Each suggestion contains:
Title — Short description (max 200 characters)
Category — Instructions, Skills, Knowledge, Guardrails, or Other
Description — Detailed recommendation (max 2,000 characters)
Suggestions can modify: agent instructions, skills, knowledge, and guardrails.
Applying Suggestions
Related sections
Monitoring — Watch runtime behavior after the agent is live.
Online-Eval — Score live conversations in production.
Reinforcement Learnings — Review and approve learning cases from real usage.
Last updated