Ctrlk
ChangelogTry UPTIQ AI
For the complete documentation index, see llms.txt. This page is also available as Markdown.
Cognition & Control

Evals

Test agent responses against scored datasets with automated LLM grading.

Evals is an evaluation framework for systematically testing agent quality on the Uptiq Platform. It allows teams to create datasets of test cases — read from an uploaded document, generated by an LLM, or added manually — run those test cases against a live agent, score each response on a 0–10 scale using an LLM scorer, and receive AI-generated suggestions for improving the agent.

Use Evals before release, Online-Eval after release, and Reinforcement Learnings to carry reviewed feedback forward.

Capability
Description
Limit

Dataset management

Create, rename, delete datasets; import test cases from an uploaded document, or generate them from the agent's instructions

100 per agent

Test case management

Question, expected answer, optional document context and attachments

50 per AI generation; up to 500 from one document

Evaluation runs

Trigger, monitor progress in real-time, cancel, view results

100 per agent

LLM scoring

0–10 scale with reasoning, 7+ = pass threshold

—

Suggested changes

AI-generated improvement recommendations with one-click apply

—

Document support

Auto-generated DOCX/XLSX/PDF test documents with preview

—

Evals is accessed from the Cognition & Control section in the Agent Builder's Agent Config sidebar.

Workflow

Manage datasets and test cases.

View runs, scoring, and suggested changes.

1

Build a dataset

Create a dataset from an uploaded document, generate one with AI, or add test cases manually.

2

Run the evaluation

Execute the dataset against the live agent in Runs.

3

Review and improve

Inspect scores, review failures, and apply suggested changes.

Datasets

A dataset is a collection of test cases for evaluating an agent. Each agent can have up to 100 datasets.

Generate Dataset

Select Generate Dataset on the Datasets tab. The Add test data & cases page opens in place of the list, in two steps.

The Evals page in Agent Config on the Datasets tab, listing datasets with their status, test case count and creation time, and the Generate Dataset button
The Datasets tab, with the Generate Dataset button.
1

Test Data & Cases

Choose where the test cases come from.

Upload a document, and the test cases are read out of it for you. The document can list the test cases, or describe the scenarios to cover; from a description, the cases are derived from the rules and constraints it states.

  • File types: .pdf, .xlsx, .xls, .md

  • Size: up to 10 MB

  • Test cases: decided by the document, up to 500

Step 1 of Add test data and cases, on the Upload file tab, with a drop zone that accepts pdf, xlsx, xls and md files up to 10 MB
Uploading a document of test cases.

Have test cases written for you from the agent's instructions.

Field
Required
Description

Number of Test Cases

Yes

1–50, defaults to 10

Additional Instructions

No

e.g., "Focus on edge cases related to pricing"

Step 1 of Add test data and cases, on the Generate with AI tab, with the Number of Test Cases and Additional Instructions fields
Generating test cases from the agent's instructions.

Generation steps:

  1. The system reads the agent's system prompt (instructions)

  2. Sends it to an LLM (Gemini 3 Flash) to generate test cases

  3. For each test case, the LLM creates: question, expected answer, and optional document content

  4. Document content is converted to actual files (DOCX, XLSX, or PDF) and uploaded to storage

  5. Test case records are created with document references

2

Review & Generate

Check the details, then generate the dataset.

Field
Description

Dataset name

Required, up to 120 characters.

Test case source

Uploaded file (with the file name) or Generated with AI.

Test cases

The number you asked for, or Decided by the document for an upload.

The new dataset's details page opens. Reading an uploaded document takes a while, so the dataset shows Extracting until its test cases are ready. You can leave the page in the meantime.

Add test cases manually

Create manual test cases one at a time in an existing dataset.

Each manual test case can include:

  • question

  • expected answer

  • optional documents

Use POST /:agentId/eval/datasets/:datasetId/items to create the test case.

Use Test Case Search to filter by question, expected answer, or document context.

How document format selection works

Document format selection is automatic.

Format
Content Type
Examples

XLSX

Tabular/spreadsheet data

Budgets, financial tables

DOCX

Narrative prose

Memos, policies, contracts

PDF

Professional documents

Invoices, resumes, statements

Dataset Views

List view displays all datasets for the agent:

  • Name — Dataset name

  • Status — See the table below

  • Test Cases — How many test cases the dataset holds

  • Created — Creation timestamp

  • Actions — Rename (pencil icon), Delete

Status
Meaning

Extracting

The uploaded document is still being read. The list refreshes on its own until it settles.

Ready

The test cases are in place and the dataset can be run.

Failed

No test cases could be read from the document. Hover over the badge for the reason, then upload a revised document as a new dataset.

Empty state: "No evaluation datasets yet. Generate one to get started."

Details view shows the dataset name as header with a "Run Evaluation" button (disabled if no test cases), plus the list of all test cases.

Test Cases

Each test case contains:

  • Question — The input prompt sent to the agent

  • Expected Answer — The ground truth / ideal response

  • Document Context (optional) — Description of attached documents (e.g., "Excel budget sheet")

  • Documents (optional) — Attached files (DOCX, XLSX, PDF) shown as clickable chips

Document Preview

Clicking a document chip opens a preview modal:

File Type
Viewer

PDF

Native document viewer

Google Docs / Office files

Embedded Google or Office Online viewer

Excel

Office Online viewer

Toggle between "Show preview" and "Show context" if document context is available. Use the "Open file" button for external download.

Runs

Starting a Run

  1. Navigate to a dataset with at least one test case

  2. Click "Run Evaluation"

  3. The system creates an eval run with status Pending

  4. Background processing starts

  5. Success toast: "Evaluation run started"

Use the Runs tab to track:

  • current status

  • progress

  • scores

  • suggestions

Each run is a card showing the dataset name, its status, when it started, and its Passed, Failed and Avg score figures. Use Search runs to find one, and select a card to open the Evaluation Run page with its summary and per-test-case results.

The Evals page on the Runs tab, with a search box and two completed run cards showing passed and failed counts and an average score
The Runs tab.

Run Execution

For each test case in the dataset:

  1. Resolve document references to signed URLs

  2. Trigger the agent via the Agent Executor (/agents/{agentId}/trigger)

  3. Pass the question + documents to the agent

  4. Extract the agent's actual answer from the response

  5. Score the response using an LLM scorer (Gemini 3 Flash)

  6. Record the result (score, reasoning, correctness, duration)

  7. Update run progress counters

Up to 3 test cases execute concurrently during a run.

Run Status

Status lifecycle
Status
Indicator
Details Shown
Available Actions

Pending

Warning badge

Waiting to start

—

Running

Animated spinner

Progress bar (completedItems / totalItems)

Stop

Completed

Green badge

Pass/fail counts, average score

View results, Apply suggestions

Failed

Red badge

Error message

—

Stopped

Red badge

Partial results

View partial results

A Stop button appears during in-progress runs. Stopping preserves already-completed results. The UI polls every 3 seconds while a run is in progress, stopping automatically when the run completes.

Scoring

Score Scale: Each test case response is scored by an LLM (Gemini 3 Flash):

Score
Verdict
Description
Result

0–3

Incorrect

Response is wrong or irrelevant

Fail

4–6

Partially correct

Some relevant content but incomplete or inaccurate

Fail

7–8

Correct

Meets the expected answer criteria

Pass

9–10

Comprehensive

Exceeds expectations with additional relevant detail

Pass

Pass threshold: A score of 7 or above marks the test case as passed.

Result Card — each completed test case displays:

  • Status Badge — Pass (green), Fail (red), or Error (red)

  • Score — "Score: X/10"

  • Duration — Execution time

  • Question — Original input

  • Expected Answer — Ground truth (muted, scrollable)

  • Actual Answer — Agent's response (or error message)

  • Scorer Reasoning — LLM's explanation of the score

Results appear progressively as each test case completes.

Completed runs display pass count (green), fail count (red), average score (0–10), and an LLM-generated summary of evaluation results.

What a failing run usually means

A failing run does not always mean the agent is broken.

Common causes include:

  • incomplete instructions

  • missing skills or knowledge

  • answers that are directionally correct but below the pass threshold

Suggested Changes

Suggestions are generated automatically when a run completes with failures or an average score below 7.

Each suggestion contains:

  • Title — Short description (max 200 characters)

  • Category — Instructions, Skills, Knowledge, Guardrails, or Other

  • Description — Detailed recommendation (max 2,000 characters)

Suggestions can modify: agent instructions, skills, knowledge, and guardrails.

Applying Suggestions

1

Open suggestions

Click "Apply to Agent" on a completed run with suggestions.

2

Review selections

A dialog opens with all suggestions and checkboxes.

Edit titles, categories, or descriptions as needed.

3

Apply changes

Select the suggestions to apply and click Apply.

Toast: "Builder agent is applying your changes — watch the chat panel on the left"

4

Continue in chat

The agent config modal closes and the Builder Agent executes the changes in the chat panel.

Related sections

  • Monitoring — Watch runtime behavior after the agent is live.

  • Online-Eval — Score live conversations in production.

  • Reinforcement Learnings — Review and approve learning cases from real usage.

Maintained by Abhishek Paul AI-assisted, human-approved

PreviousTest
NextDeploy

Last updated

Was this helpful?