Evals
Measure agent quality before release and in production.
Agents can give different answers to the same request. Evals make quality measurable, repeatable, and reviewable.
Use evals to test changes before release and monitor quality after deployment.
At a glance
Before release: Test known scenarios against expected answers.
In production: Score live executions and route uncertain results for review.
Over time: Add reviewed misses to your test dataset.
Two evaluation loops
Qore evaluates agents before and after deployment.
Pre-release evals
Before deployment
Curated datasets and expected answers
Validate planned changes
Production evaluation
After deployment
Live executions
Detect quality drift and route uncertain results
Pre-release evals test known expectations. Production evaluation tests real traffic.
Pre-release evals
In Agent Builder, an eval dataset contains test cases. Each case includes a question, expected answer, and optional documents.
Run the dataset after changing instructions, skills, knowledge, behavior, or guardrails. An LLM scorer assigns each response a score from zero to 10. Scores of seven or higher pass.
Include cases for:
Frequent requests the agent must handle well
Edge cases that expose unclear instructions or missing context
Regressions found during production monitoring
Passing results provide evidence, not certainty. Test cases cannot cover every production request.
For setup and results, see Evals.
Production evaluation
After deployment, online evaluation scores live executions in the background. It doesn't delay or change responses.
In Control Center, an evaluator defines the quality method, judge model, and confidence threshold. Results below that threshold can route to Human Review.
A low-confidence result needs a person's judgment. It doesn't necessarily mean the agent failed.
Use production results to identify drift, recurring misses, and missing test cases. Turn reviewed findings into improved instructions, skills, knowledge, or learning cases.
For organization-wide evaluation and review routing, see Evaluation & Quality. For per-agent production monitoring, see Online-Eval.
The improvement cycle
Use evaluation as a continuous control loop.
Define representative test cases and expected outcomes.
Run pre-release evals after each meaningful configuration change.
Deploy changes that meet your acceptance criteria.
Monitor live results.
Review uncertain outcomes.
Add reviewed production misses to the next test dataset.
Don't treat a model score as the final decision for high-impact workflows. Define review paths for outputs that affect customers, money, access, or compliance.
How evals fit with other controls
Evals measure output quality. Guardrails & Validation constrain what an agent can process or do. Neither replaces human review where judgment is required.
Memory & Learning closes the loop. Reviewed production feedback becomes learning cases. Evals validate those improvements before broader rollout.
Where this is configured
Evals—create datasets, run tests, and review scored results
Evaluation & Quality—define evaluators, inspect online results, and manage Human Review
Online-Eval—monitor per-agent production scoring
Reinforcement Learnings—approve learning cases from reviewed feedback
Last updated