Scoring a support copilot before it answers a customer
We built the scoreboard first, using a client's own resolved tickets.
What we found
On [n] historical tickets, the assistant matched the human answer [00]% of the time and — more importantly — correctly declined to answer [00]% of the cases it should never have touched.
Every support assistant demos well. The question that matters is what happens on the ticket nobody anticipated, and you cannot answer that with a demo.
So we start with the client's resolved-ticket history rather than the model. A sample is split into cases the assistant should answer, cases it should escalate, and cases it should refuse outright. Each one is scored the way the support lead would score it, not by string similarity.
That gives a number to argue with. It also gives a safety measure that most teams skip: the refusal rate. An assistant that is right 95% of the time and confidently wrong the other 5% is worse than no assistant, because it teaches customers to trust it.
The evaluation set is a deliverable in its own right. It belongs to the client, runs on every change, and survives whichever model is fashionable next quarter.
Draft — re-run and replace [bracketed] figures before publishing (lib/lab.ts)