Teaching an AI analyst to know when it's wrong

Company Overview

PAR Retail (opens in a new tab), part of PAR Technology, builds loyalty, payments, and customer engagement software for convenience and fuel retailers. Among its products is PAR Intelligence, a conversational analytics assistant: ask a plain-English question about your loyalty and transaction data, and it answers without anyone writing a query. Behind the scenes, the assistant turns the question into a database query, runs it, and writes back a friendly summary.

Project Overview

PAR ran the assistant as an experimental pilot: real data, real questions, a limited group of users, and no wider rollout until the team could trust the answers.

The product always looked like it worked. A traditional program that breaks looks broken: it crashes, it errors, it returns nothing. An AI assistant that is wrong looks exactly like an AI assistant that is right, with the same tone, the same polish, and the same air of authority. A confident, well-written number that happens to be false invites a business decision, and a user who hits one wrong answer rarely files a report; they just stop trusting the product.

PAR's engineering team knew the bottleneck to that rollout was not shipping features; it was knowing whether the answers were right. The key issues:

• There was no way to tell a good answer from a bad one by reading it: the dangerous failures were plausible and well written
• Explicit feedback was nearly silent: the thumbs-down button captured a small fraction of real dissatisfaction
• The expert judgment that could settle a correctness question was scarce, and it did not scale to every conversation

On top of this, the runtime itself was still evolving, so anything we measured had to survive a re-architecture.

Working with PAR's engineering team, we built the evaluation practice in layers, from the first deterministic check in late February to a full feedback loop by the end of June.

Start with checks a machine can make

Before judging whether an answer was good, we checked whether the machine behaved. The first scorers were deterministic, with no model in the loop: did the assistant come back at all, or did the request time out and return nothing? Did a question about the data actually reach the data, or was it answered without ever running a query?

These checks ran across every production trace and acted as a pre-filter. Instead of reading thousands of transcripts, a domain expert worked through the few hundred the checks flagged in a standing weekly review, recording a verdict on each with a reason: should the assistant have handed the question to the query engine, and was the top-level answer plausible? Those recorded verdicts became the ground truth the rest of the system was built on.

Graders earn trust through calibration

Reading answers does not scale, so the next layer was AI graders: models that score each trace against a written standard for questions no simple rule can settle, like whether an answer is plausible and well caveated.

The graders were built alongside the human review work, but as untrusted stubs. A new grader ran on live traffic from the start, because nothing exposes its blind spots faster, but its scores did not count until it had been checked against the expert's recorded labels and agreed with them above a bar: at least 50 labeled examples and 80 percent agreement.

We first tried an off-the-shelf prompt optimizer to align the graders, and dropped it: a tuner can reword a grader's instructions, but the changes that actually moved agreement restructured what the grader was shown and what it was asked to check, and those are changes a narrow tuner cannot propose.

A system that improves the system

With measurement in place, we pointed AI at the improvement problem itself. We call it a meta-harness: a system whose job is improving another system. A proposer agent reads the recent body of interactions in one pass and drafts candidate changes across many channels, including the assistant's prompts, its tools, its benchmarks, and the graders' own instructions. It emits proposals; it never edits anything directly.

Every proposal has to clear two gates: an automatic evaluation gate showing the measured scores improve without breaking anything, and a human approval, after which the change ships as an ordinary pull request a person applies.

Alongside it, attention directors attack the labeling bottleneck: named strategies that surface the highest-information traces for review, like a trace where a stronger model re-judged the answer and disagreed with the grader's verdict, a random sample that keeps the review honest, or the conversations a user reacted to. The expert labels the handful of traces that carry the most information instead of reviewing everything.

Changing the engine mid-flight

Midway through the work, PAR rebuilt the product's engine. The assistant had started on a managed agent framework hosted by PAR's data platform, delegating questions to the platform's natural-language query service. In May, PAR's engineers brought orchestration in-house, rebuilding the loop on an open-source AI SDK with the query service as one tool the loop calls. The rebuild ran in checkpoints over the month, so the team owned the loop, read its native traces, and evaluated its own service instead of a hosted black box.

For the evaluation system, the swap was the test of a design principle, and the questions survived it: whether the assistant handed the question to the query engine remained the thing we measured, while the mechanism underneath it changed completely. In the same stretch, the labeling pipeline moved onto the data platform as well, so expert review ran against the live traces themselves rather than copies.

Hunting the failures that look right

By June the system scored every trace, and pilot users still hit a failure the graders could not see: a confident, plausible, wrong answer. These sailed past the graders the same way they sailed past human readers, because the answers simply looked right.

The response was a research effort with a strict standard of proof: an answer that merely looked wrong stayed a suspicion, and we counted a failure mode only when we could name a check the answer cannot fake and show it firing on a real trace. Recompute the number independently from the raw data. Check the answer against itself: do its own figures add up, does a part exceed its whole, does it contradict the same question asked an hour ago. Ask whether the data needed to answer even existed, and whether the question was one the system should have answered at all.

We confirmed 24 failure modes this way and turned them into a detector suite, pinned by a 63-check regression harness so no detector could silently drift. Run over roughly 800 traces, the detectors flagged about a third, and among the flags were 35 silent failures: wrong answers that scored high and that no human had ever flagged. The suite also recalled 27 of the 30 answers earlier expert reviews had already caught.

Human reactions validate the detectors but never feed their scores, because the entire point is finding the failures humans missed. The detectors that needed no model in the loop were ported back into the deterministic layer, joining the always-on checks that run over every trace, and the findings gave the reasons recorded in the weekly review a shared vocabulary of named failure modes.

Reading behavior, not waiting for buttons

In one month of traffic, 423 conversations that asked real questions of the data produced just 7 explicit thumbs-down reactions, while the same conversations held 112 distinct dissatisfaction signals. Users say an answer is wrong in words, or doubt a plausible number and ask to see the query, or just rephrase, switch approaches, and abandon the thread.

So we stopped waiting for the button and started mining the conversations themselves on a cadence, with the definition of dissatisfaction held stable from month to month, so the trend measures the system and not a shifting rubric. Surfaced items route into two lanes: judgment calls stage into the expert review, and clear fixes go straight to the runtime.

The mining also closed a gap the research had exposed: the same business term could resolve differently from turn to turn, so we shipped a documented glossary of canonical terms, giving the assistant something to disambiguate, clarify, or decline against instead of guessing.

The evaluation loop delivered several standing capabilities:

• Everything is scored. Every production trace runs through the deterministic checks and a dozen calibrated graders, so quality is measured continuously instead of sampled when someone worries
• Hidden failures surface. The detector suite found 35 wrong answers no human had flagged, the exact failures that would otherwise reach users silently
• 16x the signal. Mining conversations for dissatisfaction surfaced 112 signals in the month the thumbs-down button captured 7, turning near-silent feedback into a steady improvement queue
• Expert judgment multiplies. The expert moved from grading flagged traces, to setting the standard the graders enforce, to approving proposed improvements, to making only the subtle correctness calls that still need a human
• The measurement outlives the stack. When the engine was rebuilt underneath it, the evaluation questions persisted while their mechanisms were swapped

From the first deterministic scorer to a running feedback loop took about four months. The loop's shape is the durable result: the deterministic checks pre-filter for the graders, the graders' verdicts direct the expert's attention, the expert's labels calibrate the graders and gate the proposals, and what the newest research discovers gets ported back into the always-on checks at the bottom. Each layer feeds the others, which is why the system keeps getting better at finding the failures that look right.