How to Evaluate an AI Feature Before You Ship It

Someone on the team ran the new feature against twenty or thirty questions they thought up over a coffee, eighteen of them looked right, and the ship decision got made. Nobody wrote the cases down. Nobody wrote down what "right" meant. When the feature misbehaves in week three there is nothing to compare against, so the argument about whether it got worse is settled by whoever remembers the demo most confidently.

That kind of check can catch obvious problems, but it does not give you a reliable baseline. A useful evaluation needs a repeatable set of cases, a clear grading rule, and a result you can recompute. Eighteen successes out of twenty sounds reassuring, but the estimate is too imprecise to carry the decision on its own. Every figure below is arithmetic you can rerun on your own numbers; none of it is a measured result from client work.

A pass rate without a sample size is not a number

If you score 18 out of 20, your best estimate of the true pass rate is 90 percent. The question nobody asks is how much that estimate could be wrong by, and the answer is: enormously.

A pass rate is a proportion estimated from a binomial sample, so it carries a confidence interval. The simple Wald interval can be unreliable near the ends of the scale. The Wilson score interval is a better default for this kind of proportion. Use the Wilson score interval instead. For an observed proportion p over n cases, at confidence level z (1.96 for 95 percent):

centre     = (p + z^2 / 2n) / (1 + z^2 / n)

half-width = z / (1 + z^2 / n)
             x sqrt( p(1-p)/n + z^2 / 4n^2 )

interval   = centre +/- half-width

Put 18 out of 20 through it and the 95 percent interval runs from 69.9 percent to 97.2 percent. Your ninety-percent feature is consistent with a true pass rate of seven in ten. It is also consistent with ninety-seven. The result leaves a wide range of possible performance, even though the point estimate sounds precise.

Precision costs cases, and it costs them quadratically

Hold the observed rate at 90 percent and grow the set. Every number in this chart comes out of the formula above with p = 0.9 and z = 1.96, so you can check all four in a spreadsheet.

What a 90 percent pass rate is actually worth

95 percent Wilson interval at p = 0.9, four eval set sizes

• 95% confidence interval
• Observed pass rate, 90%

The shape of that curve is the whole planning problem. Interval width shrinks with the square root of n, so halving your uncertainty costs four times the cases. Going from 20 to 50 cases buys you ten points of precision for thirty cases of work. Going from 120 to 400 buys you five points for two hundred and eighty. There is a knee in the curve, and for most features it sits somewhere between 100 and 400 cases.

Work backwards from the decision instead of guessing. If you need a margin of error e at pass rate p, the normal approximation gives you the case count directly:

n = z^2 x p(1-p) / e^2

p = 0.9, e = 0.05  ->  n = 3.8416 x 0.09 / 0.0025 = 139
p = 0.9, e = 0.02  ->  n = 3.8416 x 0.09 / 0.0004 = 865

So a set that pins your pass rate to plus or minus five points needs about 140 cases. Pinning it to two points needs about 865. This is an estimate for the precision of one rate. Detecting a two-point difference between two releases takes a separate power calculation, usually with many more cases.

Zero failures in twenty cases is consistent with a one-in-ten failure rate

The most misleading result an ad hoc test produces is a clean sweep. Twenty for twenty feels like proof. It is not, and there is a one-line rule for saying how much it is worth. If you observe zero failures in n trials, the upper bound on the true failure rate at roughly 95 percent confidence is 3 / n.

n = 20   ->  failure rate could be up to 15%
n = 100  ->  failure rate could be up to 3%
n = 1000 ->  failure rate could be up to 0.3%

Twenty clean cases still leave room for a meaningful failure rate. The rule of three is a quick way to put a perfect-looking score in context.

Build the set from traffic, not from imagination

Sample size is the part you can compute. Case selection is the part that decides whether the number means anything, and no amount of n rescues a set drawn from the wrong distribution.

Cases you write yourself are cases you can already imagine, which means they are systematically the easy ones. Pull from real traffic instead: logged queries, support tickets, the search box, the inbox. Then stratify deliberately, because an unweighted sample of production traffic will be dominated by the common easy case and will tell you almost nothing about the tail that generates your complaints.

A workable split for a first set:

• Head traffic, about half. The frequent, ordinary requests. This is what your headline pass rate is about.
• Known failures, about a quarter. Every case a human has complained about. These are free, they are already labelled, and they are the ones a regression will hit first.
• Edge and adversarial, about a fifth. Empty input, wrong language, prompt injection, out-of-scope questions, requests that should be refused.
• Frozen regression cases, the rest. Bugs you fixed. They exist so a fix stays fixed.

Freeze the set and version it. A set you keep adding to is a moving baseline, and a moving baseline cannot tell you whether last Tuesday's prompt change helped. Add cases in a new version, record which version each score belongs to, and never compare across versions without saying so.

Grade against a rubric only where a golden answer cannot exist

Two grading modes, and the cheap one is better wherever it applies.

A golden answer is an expected output compared by string match, JSON equality, or a rule. It is deterministic and does not require another model. Use it when the output is constrained, such as classification, extraction, routing, or structured data. For JSON output, schema checks and field-level comparisons often give you a clearer signal than a general quality score.

A rubric is for open-ended output where many answers are correct. It is a set of independent binary criteria applied by a grader, ideally a model with the criteria in its prompt: is the answer factually supported by the retrieved context, does it answer the question asked, does it decline when it should, is it free of invented citations. Score each criterion separately. A single one-to-five quality rating collapses distinct failures into one blurry number and drifts between grading runs.

The important discipline: the grader is itself a system with a pass rate. Before you trust it, have a human label a sample of maybe fifty cases and measure how often the grader agrees. If the grader often disagrees with people on important cases, improve the rubric and check it again before scaling the evaluation.

Report the interval, not the point estimate

Everything above collapses into one habit. When you report an eval result, report n and the interval alongside the rate: "88.3 percent, 95 percent CI 81.5 to 92.9, n = 200, set v4" rather than "88 percent". It takes one extra line and it prevents the two failure modes that make eval numbers useless in practice - celebrating a rise that is inside the noise, and shipping past a fall that is not.

It also changes the ship conversation for the better. "Ninety percent" invites a yes or no. "Between 70 and 97 percent on twenty cases" invites the correct question, which is what it would cost to know more precisely. That is a budget conversation with a defensible answer, and you now have the arithmetic to give it.