How do we actually know if an AI feature is any good?
One misleading running insight becomes a repeatable AI eval, and a reassuring score hides the failure that changes the release decision.
On Sunday afternoon, a runner finishes a long trail run with a few hundred metres of climbing. Earlier in the week, they did a quicker five kilometres around the usual flat loop. Their running app has both activities, and its new AI summary tells them they’ve slowed down.
The summary doesn’t sound quite right. The pace number is real, but the app has treated a longer, hillier run as evidence that the runner is losing speed. It looks odd enough that the team building the feature tries the same week again. This time the summary notices the hills and explains that the slower average came with a harder route.
Nothing was wrong with the pace number, but the first summary made a shaky call about what it meant. The second shows that the feature can get this week right, but one useful answer only proves it managed it once. AI evals help make sense of how dependable the feature is across many attempts, including which running histories trip it up and whether the next product change makes it better or merely gives the team one luckier answer.
Start with the run it got wrong
The team starts by saving the running history that produced the bad summary and writing down what a useful answer must get right. It should use the right distances, paces and elevation, compare runs that can reasonably be compared and avoid claiming that the runner’s fitness changed when the data only shows that the route did.
That running history and those success rules make one test case. A useful insight doesn’t need to match one perfect sentence, so the feature could talk about the faster flat run first or the hilly run first. Either answer can work if the facts are right and the conclusion stays inside what the running data can support.
Aren’t these just test cases?
Yes. The saved running history is still a test case, and the app still needs ordinary checks for the distance, elevation and pace. Code should also confirm that the right run data made it into the summary. An AI eval doesn’t replace any of that or give it a trendier name.
The generated summary is harder to pin to one expected result. It can answer usefully in several ways, while an answer with all the right numbers can still make a poor judgement about what they mean. Exact checks can cover the facts, but they can’t decide every reasonable way to explain the week.
The summary may not come back the same way every time either. The model can take more than one plausible path through an answer, and some product settings leave more room for that variation than others. Ordinary software tests can handle variation too, and exact checks still belong in an AI eval. For this running feature, one expected answer and one successful attempt still aren’t enough to show that it behaves well.
Keep adding running histories like this and the team has an eval suite. The eval is the whole exercise of running them, judging the summaries and comparing versions. Repeated attempts belong in this one because the running summary has already varied, although they won’t be necessary for every feature with stable outputs.
Run the same week more than once
The first running history has already produced two different answers. One blamed the runner’s pace and the other noticed the hill, even though both received the same input. Running the case five times shows whether the useful answer appears consistently or only now and then. If three attempts account for the elevation and two still blame the pace, saying the case passed would hide the instability the team is trying to understand.
Those five attempts only mean something if they came from the same version of the feature. The AI model matters, but so do the instructions, the run data and the product code that turns an answer into the insight on screen. Any result the team records describes that whole setup. Once something meaningful changes, the team has a new version to compare with the old one.
One strange week isn’t enough
Five attempts at the same strange week still leave most of the running world untested. The team adds steady weeks on the same route, a first long run, intervals with slow recovery sections, treadmill runs with no elevation and messy recordings where the GPS has clearly had a strange afternoon. It also keeps examples where the feature has confused a hill with declining fitness, because those are mistakes the team already knows are worth challenging. For this example, it keeps adding and balancing those kinds of weeks until the set contains 100 running histories.
Those 100 running histories become the eval set. Choosing them matters as much as counting them. A hundred copies of the easy flat week would create a larger spreadsheet while saying very little about the histories that make the feature work harder.
Running each history five times gives the team 500 insights. That’s far more evidence than one good answer, but only after every attempt has been marked against the same rules.
Five runs isn’t a rule for every eval. It gives this example useful arithmetic, while the amount of testing still needs to match the product claim, the variation in the results and what a bad insight could change for the runner. An early idea may need a small set. A claim about many kinds of runners, routes and recording quirks needs broader evidence.
Who decides whether an insight passed?
Code can check whether the distance and pace in each insight match the underlying run data, or whether the feature has invented a personal best that never happened. Those exact checks can mark part of all 500 attempts without asking anyone to debate them.
Code can confirm the exact numbers, but the team still needs to decide whether the insight makes a fair connection between them. A person can read the run history and ask whether the conclusion is supported, whether important context disappeared and whether the result would leave a runner with the wrong impression. The guide used to make those decisions is often called a rubric, which is really just a more specific version of what good means for this feature.
Reading every insight by hand is manageable at first. At 500 attempts, it’s already a job of its own. Another AI model can apply the same rubric to more results and act as a model grader, sometimes called an AI judge. That can make the work much faster, but it also means one AI system is measuring another. The grader can prefer a certain style, miss a subtle unsupported claim or change its judgement when the same choices are presented differently.
A model grader doesn’t remove the judgement. People who understand the running data still need to mark a sample and compare their decisions with it. When the marks disagree, the team checks the rubric, the grader and the borderline answer. The rubric may be vague, the grader may be inconsistent or informed people may genuinely see a grey area differently.
Once the team is comfortable with the marking, it still needs to turn 500 marks into a score. Every attempt counts equally towards the overall pass rate in this eval, and each result stays tied to its running history so the team can inspect where the failures gather. Keeping that method fixed lets the team compare versions without changing the way it counts at the same time.
If the current service passes 420 of the 500 attempts, its score is an 84 per cent pass rate. It means this version passed 420 attempts on these 100 histories under these rules. It doesn’t mean the service understands 84 per cent of all running, and it definitely doesn’t make every future insight 84 per cent trustworthy.
The better score still gets the hills wrong
With all 500 attempts marked, the team can put the current service beside a proposed new version. The new version passes 445, which gives it 89 per cent. The current version passes 420, which gives it 84 per cent. If the team stopped at the big number, the new version would look like the obvious winner.
When the team separates the mixed-terrain histories, they tell a different story. Ten examples combine flat running with hills, so five attempts at each one give the team 50 results to inspect. The new version turns the terrain change into a claim about declining pace in 20 of them. The current version does it in five.
Those totals are still correct, but they’re answering a blunter question than the release decision needs. The proposed version passed more attempts outside the mixed-terrain slice while becoming much worse at the exact situation that made the team suspicious in the first place. The team can now investigate whether its new instructions push too hard for a simple trend, whether useful elevation context is getting lost or whether the grader is rejecting a valid way of describing the week.
Someone had to think of including those mixed-terrain histories before the eval could reveal that pattern. The team can also get too good at passing its own test if it keeps changing the feature while staring at the same examples. Saving some histories until the end gives the proposed version a cleaner challenge. People usually call this a held-out set, although it still can’t represent every running week the feature will meet later.
Would the team release it?
Not yet. The 89 per cent score is still useful because it shows that the proposed version passed more of this eval under the same rules. The mixed-terrain results also show that it became much worse at the kind of week that started the whole investigation. A higher score can’t make that trade-off disappear.
The team might change the instructions, keep the current version or limit the new insight until it understands the hill failures. No universal passing score can make that choice for every AI product. What counts as good enough depends on the job and what a bad result could change for the person relying on it.
If the feature reaches runners, they’ll record weeks nobody thought to include in the eval. Once the team understands a new failure, it can become another test case, with care around consent, access and personal information in the running history. The prepared set can keep improving without pretending it’ll predict everything that happens next.
Until the next version can keep the overall gains without turning hills into lost fitness, the team holds it back. After each fix, it can rerun the mixed-terrain cases and see whether the summary now accounts for the hills without losing what it improved elsewhere.