Our AI Agent Passed the Answer Test—and Still Took the Wrong Action

“Add a mobility session to Friday” sounds like a simple request. A response-only grader might reward a polite sentence promising that Friday is updated. But the user wanted a changed plan. A useful test records the starting plan, the requested change and the expected final state. If the model writes a friendly answer but the plan remains unchanged, the task failed. If it changes Thursday instead, the task also failed.

That separation helps the interface too. A proposal card is evidence that the assistant understood an intention. A saved-state response is evidence that an application operation completed. Neither should impersonate the other. The frontend should use the operation result to refresh the relevant resource, rather than infer success from reassuring prose. Evaluation should make the same distinction.

One of the most valuable lessons in our evaluation records is that a test can stay green while exercising an older path. A previous coach architecture and the currently deployed agent runtime are different test targets. Running an excellent suite against the wrong engine still leaves the real product exposed.

I would begin a release checklist by recording which runtime, model configuration and tool registry the replay uses. This does not require publishing internal configuration. It requires an explicit relationship between the application version being reviewed and the behaviour being tested. The same applies to datasets: a stored assistant answer is not a valid replacement for the original user request. A perfectly replayed mistake in dataset construction is still a mistake.

The documented coach evaluation flow runs real model decisions while replacing tool effects with fixtures. That lets the agent select a tool and construct arguments without touching a real account. The fixture can say a fictional plan exists, a save succeeded or an operation failed. The test observes what the agent does with those conditions.

This is powerful precisely because its scope is limited. It tests decisions under supplied conditions. It does not establish that a real database query is correct, a permissions service is configured properly or a mobile cache refreshes. Those need separate integration checks. I prefer a clear boundary to a broad “AI tested” badge that hides which parts were simulated.

Consider an invented two-turn conversation. First, the assistant proposes adding a short session to day three. Then the user replies, “Yes, add it.” The test should preserve the pending proposal and enough history to resolve “it.” Otherwise it is measuring how well the agent guesses missing context, rather than how it handles approval.

The expected behaviour could be: choose the workout specialist, call the permitted edit operation with day three, wait for its result and describe only the change that succeeded. A second version returns a save error. A third contains two plausible proposals and should require clarification. These are three different behaviours, not interchangeable paraphrases of one happy path.

Scroll the table horizontally to see all columns.

A practical scorecard can check routing, required and forbidden tools, argument values and ordering. Order matters when a read or preview must precede a write. It also matters whether a test requires an exact sequence or merely an ordered subsequence that allows harmless extra reads. Choose that rule deliberately.

Hard failures should remain visible even if the prose is excellent. An unauthorised write, an exposed internal identifier or an action before required approval cannot be averaged away by several easy questions. Softer quality measures can still track coverage and correctness over the dataset. Our recorded workflow makes that distinction; it does not claim that a single combined percentage captures every kind of risk.

Retrying a failed case can help distinguish a transient provider failure from a persistent problem. Keeping the better attempt, however, changes what the reported result means. It no longer describes first-attempt reliability. Hiding that distinction would make a prompt change look more dependable than users actually experience.

The team’s documented workflow also includes repeat mode without that favourable retry selection. It is intended to reveal how often a case succeeds across repeated runs. I would keep both views: scheduled regression detection and deliberate reliability measurement. A small sample is still uncertain, so a borderline difference deserves more evidence rather than a confident percentage attached to one run.

A synthetic account must not inherit a previous case’s conversation state. A missing fixture must not silently fall through to a real write. Cleanup and isolation are part of evaluation correctness, not optional housekeeping. They matter particularly when the agent carries pending selections or remembers earlier turns.

I also want traces to be useful without turning the evaluation system into a second store of sensitive conversations. A fictional scenario can reproduce the shape of a failure without preserving a person’s meals, health history or messages. Record the reason for failure and the relevant tool outcome. Collect raw content only when it is necessary and appropriately controlled.

I would start with one concrete failure, create an independent reproduction and define the required outcome. Then I would compare the old and new behaviour on the same cases, including nearby tasks that the change might accidentally damage. A prompt that fixes one example by refusing everything has not improved the product.

My release questions are practical: is the runtime the right one, are tool effects isolated, are failures classified, does a retry change the headline result, and can someone inspect an individual case? The documented schedule need not be called daily simply because an old filename says so. Accuracy in the explanation is part of the engineering discipline.

Can a language-model judge replace deterministic checks? It can help assess nuanced text, but an expected argument, forbidden operation or stored outcome is often better checked directly. A second model’s confidence does not establish that an application action happened.

Do fixtures make the suite unrealistic? They deliberately remove one source of uncertainty so the decision layer can be examined. Pair them with integration tests for real operations and a small, controlled end-to-end check. The layers complement each other.

Does a passing suite mean the coach is safe for every situation? No. It establishes behaviour for the covered scenarios under the tested conditions. New user tasks and model changes create new cases to learn from. The useful result is an inspectable feedback loop, not a permanent certificate.