Evals: FAQs & Troubleshooting
Answers to common Evals questions, and fixes for missing scores, failed runs, and empty review queues.
This page answers common questions about evaluators, Online Eval, and Human Review. Use the troubleshooting entries when a score is missing, a run fails, or a review task doesn't reach anyone.
FAQs
Evaluators
Who can create evaluators and see results?
Build evals: Developer, Governance Manager, Account Admin, and Organisation Admin. View online evals: Approver, Developer, Governance Manager, Auditor, Account Admin, and Organisation Admin. See Roles.
Is an evaluator created per agent?
No. Evaluators are defined at the account level. Use Agent enablement to choose which agents each evaluator scores, so one evaluator can serve every agent that enables it. See Evaluators.
What's the difference between Single judge and Jury?
Single judge scores each output with one model. Jury scores it with 2 to 5 judge models, which can come from different providers, and combines their votes into one weighted verdict. See Choose a framework.
Which evaluation methods are there?
Answer Relevancy, Faithfulness, Safety, and Task Completion. See Create an evaluator.
What does Faithfulness measure?
How much of what the agent said is supported by the context it was given: retrieved context, tool results, and what the user said. It judges support, not usefulness or correctness, so a true claim that isn't in the context counts as unsupported. See Faithfulness.
What does the judge see?
A transcript of the conversation: every message, the documents on each turn, each tool call's arguments and results, and rich replies rendered as text. Long tool results are shortened, and a conversation too long for the judge is trimmed to whole turns. See What the judge scores.
Scores and review
Why did a passed result go to Human Review?
Routing depends on confidence, not the verdict. A result routes to Human Review when the judge's confidence is below the evaluator's threshold and the evaluator has Route low-confidence results to human review on. A failed result with confidence at or above the threshold is auto-accepted. See Online Eval.
Is every conversation scored?
Only the sampled share. Each agent's Sampling sets the chance that a live conversation is evaluated; it defaults to 100%. See Agent enablement.
Does scoring cost credits?
Yes. Online Eval scoring is billed on its own, as OnlineEval on the Features tab of Credit Usage, and isn't counted in the agent's own row. Lower Sampling on high-traffic agents to keep judge costs down. See Credit Usage.
How are Human Review tasks assigned?
Through teams, not directly to people. A reviewer who isn't on a team has no assigned tasks. See Human Review.
How do I export evaluation results?
Online Eval has no export button. Open Observe › Export and choose the Eval Results dataset: scores, verdicts, reasoning, evidence, and human overrides. See Export results.
Troubleshooting
Missing scores
An agent's conversations aren't being scored
Check all three: evaluators are assigned to the agent, the agent's Online eval toggle is on, and Sampling isn't set too low. Assigning evaluators doesn't start scoring on its own. See Assign evaluators to an agent.
An agent shows Unsupported instead of the Online eval toggle
That agent's type doesn't support Online Eval yet. You can still pre-assign evaluators with Edit. See Agent enablement.
A score is missing for a conversation
Open the Runs tab, which lists every run, including those that produced no score. A Failed or Skipped run never appears on Scores. See Runs.
A run shows Failed
Hover over the status for the reason:
Judge model unavailable: <model>: the evaluator's judge model isn't available.Evaluation method not found: the evaluator's method no longer exists.Evaluation timed out: the run stayed Pending or Running for more than two hours.Evaluation failed: scoring hit an error.
See Runs.
A run shows Skipped
Its evaluator was deleted before scoring started, or the conversation had nothing to score. See Runs.
A score has no reasoning or evidence
The judge returned none. Reasoning then reads The judge did not return a justification for this score., and no Evidence list is shown. See Judge reasoning and evidence.
Evaluators and review
New evaluator is disabled
Hovering shows You don't have access to create or change evaluators. Creating an evaluator needs the Build evals capability: Developer, Governance Manager, Account Admin, or Organisation Admin. See Create an evaluator.
Create stays disabled in the new evaluator dialog
Set a name, a method, and the judge model, or 2 to 5 jury models. See Create an evaluator.
Low-confidence results never reach Human Review
The evaluator's Route low-confidence results to human review is off by default. While it's off, low-confidence results are scored and auto-accepted. Turn it on in the evaluator. See Create an evaluator.
A reviewer sees no tasks under My tasks
Tasks are assigned through teams. Add the reviewer to a team in Admin › Teams, or check Unassigned. See Human Review.
Related
Evaluate agent quality: every Evals task
Evals in Agent Builder: test datasets before release
Maintained by Abhishek Paul
AI-assisted, human-approved
Last updated
Was this helpful?