What actually runs on one GPU
Comparing small open models against a frontier API on three business tasks.
What we found
For classification and extraction, a small self-hosted model came within [0] points of the frontier API at roughly [00]% of the cost per million tokens. For open-ended drafting, it did not come close.
Not every task needs the largest available model. Classification, extraction and routing are narrow, repetitive, and — crucially — easy to measure, which makes them the best candidates for a smaller model you host yourself.
We ran three representative business tasks: categorising inbound enquiries, pulling line items out of supplier documents, and drafting a customer-facing reply.
The first two are where self-hosting pays. The last one is where it does not — quality fell off in a way that a cost saving does not justify.
The practical conclusion is that 'which model' is the wrong question. The right one is which tasks are narrow enough to own outright, and which are worth renting from an API.
Draft — re-run and replace [bracketed] figures before publishing (lib/lab.ts)