Building 50 evaluation questions that predict production quality
Teams that measure tend to measure the easy cases, because those are the ones where someone already knows the answer. The questions that will cost you money are the ones nobody thought to include.
What happened
A team builds an eval set of fifty questions, scores well, ships, and then watches user satisfaction drift down for two months without a single metric moving. The set was not wrong; it was unrepresentative. It sampled the questions the builders already knew the answers to, which are by construction the ones the system handles best.
Why it matters for deployment teams
An eval set is a statistical instrument, and its only job is to predict behaviour on traffic you have not seen. A set built from the team's imagination will overstate quality on exactly the dimensions the team understands, and stay silent about the ones they do not: jargon, malformed queries, questions spanning two documents, and questions the corpus cannot answer at all.
How to build it
Source from reality wherever you can — support tickets, search logs, the questions sales gets asked. Where you cannot, get someone who does the job daily to write them, not the person who built the system. Then enforce a composition. Roughly forty should be answerable, drawn across difficulty levels. Five must be unanswerable from the corpus, to test whether the system refuses instead of inventing. Five should have a convincing distractor document that looks relevant and is not.
Keeping it honest
Refresh the set quarterly and never let the model see it during development, or you will tune to it. Log a sample of live queries each week and check whether they resemble your set; when they stop resembling it, the set has expired. And treat a regression of more than five points on context recall as a build failure, because a metric nobody enforces is a metric that will be ignored the week before a deadline.