Research into practice

When AI scores mislead

A better score can conceal an unchanged mistake. How I would separate agent quality, case mix and evaluator changes before a business decision.

Mariya Valui · Technology Entrepreneur · 5 min read

Published: 19 October 2026Updated: 19 October 2026

A support agent gets a better quality score. The temptation is to expand its workload. But what improved: the agent, the questions it received, or the way somebody marked its answers?

That distinction matters before a score becomes a business decision. A higher number can justify more autonomy, a larger customer-facing rollout or confidence that a recurring problem has been fixed. If the comparison changed underneath it, the decision rests on a moving ruler.

The evaluator is part of the measurement

A preliminary working paper by Liu Zhang and Mark Esposito, submitted on 4 October 2026, studies 2,611 scored production interviews from one voice-and-video agent. Overlapping reviewers let the authors distinguish reviewer severity from changes between batches. They find that reviewer composition obscured the performance trend; adjusting for reviewer effects changed its interpretation. They also examine how evidence ages and compare evaluation movement with an operational signal. This is one deployment, not a universal estimate of agent reliability. The design does not establish that evaluation caused improvement. Read the working paper, version 1.

My practical reading is that a score needs an identity. What system produced the work? What cases were selected? Which criteria and evaluator produced the verdict? Without those details, “quality improved” leaves too many possible explanations open.

A better score, an unchanged mistake

Consider an illustrative business example, not an experiment I have run. An AI assistant prepares replies about subscription renewals. A customer asks why the amount increased. The assistant writes a clear explanation, uses a friendly tone and cites the pricing page. However, it misses a discount recorded in the customer's contract.

A rubric focused on clarity could mark the reply highly. A rubric requiring the correct contractual amount would fail it. Neither verdict tells us much until we know what the business needed. Polished language did not resolve the customer's question.

Now suppose the second evaluation contains more straightforward renewals and fewer disputed discounts. Even with the same rubric, its average could improve without any improvement on the difficult cases. A useful comparison needs to expose that change rather than bury it inside one score.

Keep a comparison you can explain

I would retain a small set of reference cases when changing a workflow. They should include an ordinary renewal, a conflicting contract term, a missing record and a customer whose identity has not been established. These examples are proposed test design, not published performance results.

Run the old and proposed configurations against those same cases. Preserve the inputs and record the output, tool results and judgment. If a new rubric is necessary, mark some shared outputs under both versions. This does not eliminate disagreement. It makes disagreement inspectable: a reviewer can point to the criterion that changed instead of arguing over an unexplained average.

The reference set should not become the whole definition of quality. A system can get better at familiar examples while missing new situations. I would keep incoming failures separately and decide which ones deserve a permanent regression check. That preserves a stable comparison while allowing the evaluation to learn from actual work.

A good answer and a completed action are different tests

Anthropic's engineering guide distinguishes an agent's transcript from the resulting state of its environment and describes code-based, model-based and human graders. The useful point for a founder is that different questions need different evidence. A judgment about tone cannot substitute for checking whether the intended action happened. Demystifying evals for AI agents.

In the renewal example, I would separate three outcomes. Did the reply use the correct contract? Did it explain the amount understandably? If a change was authorised, does the account now contain the correct renewal state? Those checks can fail independently. Combining them too early loses the information needed to repair the workflow.

The same distinction helps with stopping. A missing contract should produce a visible unresolved item, not a confident substitute from a public pricing page. That is a useful completion state for investigation, even though it is not a completed customer resolution.

Decide what a failure changes

Evaluation has limited practical value if every failure ends as another dashboard entry. Before running a review, I would define its possible consequences: correct a source, improve retrieval, change a tool interface, narrow an action or add a check before the next release.

For example, “wrong renewal amount” is a symptom. If the contract was never retrieved, rewriting the answer prompt may leave the real defect intact. If the correct contract was retrieved but ignored, the investigation is different. The evidence should let the next agent locate that distinction without reconstructing the entire conversation.

I would close the issue against a named revision and the failed example, then check nearby cases. Passing the one repaired case is evidence of a repair; it is not evidence that every renewal scenario works. This is my proposed operating approach, not a claim that I tested the study's agent.

When to refresh the evidence

A date alone cannot say whether a result is still useful. A change to the model, source documents, customer population or permissions may invalidate a recent check. Conversely, a stable, narrow calculation may remain comparable longer. I would attach explicit triggers to the result: what change requires the relevant checks to run again?

The decision I want an evaluation to support is concrete: can this version handle this kind of task under these conditions? If the answer is uncertain, the next step should identify the missing evidence. That is more useful than a larger score whose meaning nobody can explain.