> For the complete documentation index, see [llms.txt](https://docs.uptiq.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.uptiq.ai/ai-workbench-qore/core-concepts/evals.md).
# Evals
Measure agent quality before release and in production.
Agents can give different answers to the same request. Evals make quality measurable, repeatable, and reviewable.
Use evals to test changes before release and monitor quality after deployment.
#### At a glance
* **Before release:** Test known scenarios against expected answers.
* **In production:** Score live executions and route uncertain results for review.
* **Over time:** Add reviewed misses to your test dataset.
## Two evaluation loops
Qore evaluates agents before and after deployment.
| Loop | When it runs | What it uses | Purpose |
| ------------------------- | ----------------- | ------------------------------------- | ------------------------------------------------ |
| **Pre-release evals** | Before deployment | Curated datasets and expected answers | Validate planned changes |
| **Production evaluation** | After deployment | Live executions | Detect quality drift and route uncertain results |
Pre-release evals test known expectations. Production evaluation tests real traffic.
## Pre-release evals
In Agent Builder, an eval dataset contains test cases. Each case includes a question, an expected answer, and optional documents. Create a dataset three ways:
* **Upload a document.** The test cases are read out of it, up to 500. The document can list the cases, or describe the scenarios to cover.
* **Generate with AI.** An LLM writes the test cases, between 1 and 50.
* **Add them manually.**
Run the dataset against the agent. An LLM scorer assigns each response a score from 0 to 10, with its reasoning. Scores of 7 or higher pass. The results come with suggested changes to the agent.
For setup and results, see [Evals](/ai-workbench-qore/console/agent-builder/test/evals.md).
## Production evaluation
After deployment, online evaluation scores live executions in the background. It doesn't delay or change responses.
In **Control Center → Evals**, an **evaluator** defines how executions are scored. Evaluators are defined at the account level, so one evaluator can serve any agent it is assigned to.
| Setting | What it sets |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Framework** | **Single judge** scores with one model. **Jury** scores with 2 to 5 models and takes a weighted vote. A jury's models can come from different providers. |
| **Evaluation method** | **Answer Relevancy**, **Faithfulness**, **Safety**, or **Task Completion** |
| **Judge model(s)** | The model or models that score. The picker lists the models your organization has enabled in AI Gateway. |
| **Confidence threshold** | From 0 to 100%, default `50%`. A result below the threshold is flagged, whatever its rating. |
| **Human Review routing** | Off by default. When on, a flagged result becomes a task in **Human Review**. |
**Faithfulness** rates how much of what the agent said is supported by the context it was given: retrieved context, the results of the tools it called, and what the user said. The score is the share of the response's factual claims that the context supports.
To score an agent, assign evaluators to it in **Agent enablement** and turn on its **Online eval** toggle. **Sampling** sets the chance that each live conversation is evaluated, `100%` by default.
**Online Eval** shows the results in two tabs. **Scores** lists each scored execution, with the points the judge based its verdict on and a link to the conversation it judged. **Runs** lists each evaluator's run with its status: **Pending**, **Running**, **Completed**, or **Failed**, with the reason a run failed. Online-eval model usage appears on its own line in Credit Usage.
For organization-wide evaluation and review routing, see [Evals](/ai-workbench-qore/control-center/evals.md). For per-agent production monitoring, see [Online-Eval](/ai-workbench-qore/console/agent-builder/monitor/online-eval.md).
## How the loops connect
A reviewer in **Human Review** records a verdict for each low-confidence result, and can pass the findings to the agent team to update regression tests, instructions, or knowledge. You can add reviewed production misses as test cases in a pre-release dataset.
## How evals fit with other controls
Evals measure output quality. [Guardrails & Validation](/ai-workbench-qore/core-concepts/guardrails-and-validation.md) constrain what an agent can process or do. Neither replaces human review where judgment is required.
[Memory & Learning](/ai-workbench-qore/core-concepts/memory-and-learning.md) closes the loop. Reviewed production feedback becomes learning cases. Evals validate those improvements before broader rollout.
## Where this is configured
* [Evals](/ai-workbench-qore/console/agent-builder/test/evals.md)—create datasets, run tests, and review scored results
* [Evals](/ai-workbench-qore/control-center/evals.md)—define evaluators, inspect online results, and manage Human Review
* [Evaluators](/ai-workbench-qore/control-center/evals/evaluators.md)—frameworks, evaluation methods, and agent enablement
* [Online-Eval](/ai-workbench-qore/console/agent-builder/monitor/online-eval.md)—monitor per-agent production scoring
* [Reinforcement Learnings](/ai-workbench-qore/console/agent-builder/monitor/reinforcement-learnings.md)—approve learning cases from reviewed feedback
***
{% columns %}
{% column width="83.33333333333334%" %}
Maintained by Abhishek Paul
AI-assisted, human-approved
{% endcolumn %}
{% column width="16.666666666666664%" %}
{% endcolumn %}
{% endcolumns %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.uptiq.ai/ai-workbench-qore/core-concepts/evals.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
