Vilozi Engineering
· 6 min read
Frontier models are remarkable generalists. Most of the work inside a trades business isn't general. It's the same narrow judgement made thousands of times: what kind of job is this, how urgent is it, which crew fits it, which lines belong on the quote. For some of those, a small model tuned on industry work does better.
What "better" means
We judge a model on three things for each task: how often it's right, how fast it answers and what it costs per answer. A frontier model might be slightly more accurate at classifying a job, but if a small model is nearly as accurate, ten times faster and a fraction of the cost, it wins for a task that runs on every incoming call.
Where small models win
The tasks that suit them share a shape. The input is short, the output is one of a known set of answers or a short structured record, and there are many real, permitted examples to learn from. Sorting an incoming request into a type of job, pulling a scope of work out of a description and matching a request to the right crew all fit.
Open-ended work doesn't. Writing a careful reply to an unusual customer question, or reasoning through a complicated quote, still goes to a frontier model.
How we tune them
We start from an open model and tune it on examples built from real job data we're permitted to use, with every answer checked by a person. Customer data never goes to outside providers for training. The tuned model then faces the same evaluation set as every frontier model for that task. It only takes over if it wins.
Keeping them honest
Narrow models can drift when the work changes: a new service, a new season, a new kind of request. So each one keeps running against fresh evaluation cases, and its answers are compared with a frontier model's on a sample of live traffic. When the two start to disagree, the task goes back to the frontier model until the small one is retuned.
The point
The goal isn't to replace frontier models. It's to give each task the model that does it best. Sometimes that's the largest model available. Often, for the narrow jobs that make up most of a working day, it's a small one we built for exactly that.
Vilozi Engineering
Notes from the Vilozi engineering team on the open problems we work on.
