How to Evaluate an AI Feature Before You Ship It

Someone on the team ran the new feature against twenty or thirty questions they thought up over a coffee, eighteen of them looked right, and the ship decision got made. Nobody wrote the cases down. Nobody wrote down what "right" meant. When the feature misbehaves in week three there is nothing to compare against, so the argument about whether it got worse is settled by whoever remembers the demo most confidently.

That is not a low-quality eval. It is not an eval at all. An eval is a frozen set of cases, a stated grading rule, and a number you can recompute on demand. Manual testing produces none of those, and the number it appears to produce - eighteen out of twenty, ninety percent - is so imprecise that it cannot support the decision it was used to make. Every figure below is arithmetic you can rerun on your own numbers; none of it is a measured result from client work.

A pass rate without a sample size is not a number

If you score 18 out of 20, your best estimate of the true pass rate is 90 percent. The question nobody asks is how much that estimate could be wrong by, and the answer is: enormously.

A pass rate is a proportion estimated from a binomial sample, so it carries a confidence interval. The interval most stats packages give you by default is the Wald interval, and it is a poor choice near the ends of the scale, which is exactly where a good AI feature lives. Use the Wilson score interval instead. For an observed proportion p over n cases, at confidence level z (1.96 for 95 percent):

centre     = (p + z^2 / 2n) / (1 + z^2 / n)

half-width = z / (1 + z^2 / n)
             x sqrt( p(1-p)/n + z^2 / 4n^2 )

interval   = centre +/- half-width

Put 18 out of 20 through it and the 95 percent interval runs from 69.9 percent to 97.2 percent. Your ninety-percent feature is consistent with a true pass rate of seven in ten. It is also consistent with ninety-seven. You learned almost nothing, and you learned it in a form that sounds authoritative.

Precision costs cases, and it costs them quadratically

Hold the observed rate at 90 percent and grow the set. Every number in this chart comes out of the formula above with p = 0.9 and z = 1.96, so you can check all four in a spreadsheet.

What a 90 percent pass rate is actually worth

95 percent Wilson interval at p = 0.9, four eval set sizes

• 95% confidence interval
• Observed pass rate, 90%

The shape of that curve is the whole planning problem. Interval width shrinks with the square root of n, so halving your uncertainty costs four times the cases. Going from 20 to 50 cases buys you ten points of precision for thirty cases of work. Going from 120 to 400 buys you five points for two hundred and eighty. There is a knee in the curve, and for most features it sits somewhere between 100 and 400 cases.

Work backwards from the decision instead of guessing. If you need a margin of error e at pass rate p, the normal approximation gives you the case count directly:

n = z^2 x p(1-p) / e^2

p = 0.9, e = 0.05  ->  n = 3.8416 x 0.09 / 0.0025 = 139
p = 0.9, e = 0.02  ->  n = 3.8416 x 0.09 / 0.0004 = 865

So a set that pins your pass rate to plus or minus five points needs about 140 cases. Pinning it to two points needs about 865. If somebody wants to detect a two-point regression next quarter, that is the budget, and it is worth knowing before you promise the dashboard.

Zero failures in twenty cases is consistent with a one-in-ten failure rate

The most misleading result an ad hoc test produces is a clean sweep. Twenty for twenty feels like proof. It is not, and there is a one-line rule for saying how much it is worth. If you observe zero failures in n trials, the upper bound on the true failure rate at roughly 95 percent confidence is 3 / n.

n = 20   ->  failure rate could be up to 15%
n = 100  ->  failure rate could be up to 3%
n = 1000 ->  failure rate could be up to 0.3%

Twenty clean cases are compatible with a feature that fails one request in seven. That is the rule of three, and it is the fastest way to end a meeting where a perfect score is being treated as a green light.

Build the set from traffic, not from imagination

Sample size is the part you can compute. Case selection is the part that decides whether the number means anything, and no amount of n rescues a set drawn from the wrong distribution.

Cases you write yourself are cases you can already imagine, which means they are systematically the easy ones. Pull from real traffic instead: logged queries, support tickets, the search box, the inbox. Then stratify deliberately, because an unweighted sample of production traffic will be dominated by the common easy case and will tell you almost nothing about the tail that generates your complaints.

A workable split for a first set:

• Head traffic, about half. The frequent, ordinary requests. This is what your headline pass rate is about.
• Known failures, about a quarter. Every case a human has complained about. These are free, they are already labelled, and they are the ones a regression will hit first.
• Edge and adversarial, about a fifth. Empty input, wrong language, prompt injection, out-of-scope questions, requests that should be refused.
• Frozen regression cases, the rest. Bugs you fixed. They exist so a fix stays fixed.

Freeze the set and version it. A set you keep adding to is a moving baseline, and a moving baseline cannot tell you whether last Tuesday's prompt change helped. Add cases in a new version, record which version each score belongs to, and never compare across versions without saying so.

Grade against a rubric only where a golden answer cannot exist

Two grading modes, and the cheap one is better wherever it applies.

A golden answer is an exact expected output, compared by string match, JSON equality, or a regex. It is deterministic, free to run, and needs no model. Anything with a constrained output space should be graded this way: classification, extraction, routing, structured generation, tool selection. If your feature emits JSON, your eval is an equality check, and you should stop looking for something cleverer.

A rubric is for open-ended output where many answers are correct. It is a set of independent binary criteria applied by a grader, ideally a model with the criteria in its prompt: is the answer factually supported by the retrieved context, does it answer the question asked, does it decline when it should, is it free of invented citations. Score each criterion separately. A single one-to-five quality rating collapses distinct failures into one blurry number and drifts between grading runs.

The important discipline: the grader is itself a system with a pass rate. Before you trust it, have a human label a sample of maybe fifty cases and measure how often the grader agrees. If agreement is 85 percent, your eval has a plus or minus fifteen point fog on it that no sample size will clear, and the fix is a better rubric rather than more cases.

Report the interval, not the point estimate

Everything above collapses into one habit. When you report an eval result, report n and the interval alongside the rate: "88.3 percent, 95 percent CI 81.5 to 92.9, n = 200, set v4" rather than "88 percent". It takes one extra line and it prevents the two failure modes that make eval numbers useless in practice - celebrating a rise that is inside the noise, and shipping past a fall that is not.

It also changes the ship conversation for the better. "Ninety percent" invites a yes or no. "Between 70 and 97 percent on twenty cases" invites the correct question, which is what it would cost to know more precisely. That is a budget conversation with a defensible answer, and you now have the arithmetic to give it.