The open model interregnum

The Open Model Interregnum: Why Fit Beats Horsepower When You Build With AI

Choosing a language model for a real product is no longer a question of which one tops the benchmark. The teams we work with are learning that the model closest to the top of a leaderboard is often not the one that serves their users best. The deciding factor is fit: how well a model is tuned to the task, the prompts, and the personality of the system it lives inside. That distinction is quietly reshaping how we advise clients on the open versus closed question.

The moment the ground shifted

For most of the last two years, the practical answer to "which model should we build on" was simple. You reached for a frontier closed model from one of the large labs, paid by the token, and got dependable quality. Open weight models were interesting for research and cost control, but few teams trusted them with production work that mattered.

That assumption is aging fast. The open ecosystem now ships models that are genuinely competitive on capability. Hosting providers will run those weights for you, so you no longer need a cluster in the basement to use them. The result is an interregnum: a period between one settled order and the next, where the old rule of thumb no longer holds and the new one has not yet set.

We see clients caught in exactly this gap. They know closed models still lead on the hardest reasoning. They also know their monthly bill is climbing, their dependence on a single vendor is deepening, and the open alternatives are no longer a joke. The honest answer to "should we switch" has become "it depends," and the details of what it depends on are where the value is.

What actually breaks when you switch

The instructive part is what happens when a team moves an existing assistant from a closed model to an open one without changing anything else. In our experience, the result is rarely a graceful downgrade. It is a sharp, confusing drop in usefulness.

The assistant that felt sharp yesterday starts missing instructions. It ignores tools it used to call correctly. It burns tokens and real money while producing work no one can ship. The frustrating thing is that you cannot immediately tell whether the model is the problem or your own configuration is. Two impressive models, tested on paper, can behave completely differently inside the same system, and the reason usually has nothing to do with raw intelligence.

The gap between leading models is smaller than the gap in how they are tuned. Prompts, parameters, the default persona baked in on the model side, the way each one interprets a system instruction. All of that varies, and all of it decides whether a given model lands or falls flat for your specific workload. A model that dazzles in a chat window can quietly fail inside an agent that has to plan, call tools, and hold a long thread of context.

Fit is an engineering problem, not a shopping decision

This is the point we keep returning to with clients. Selecting a model is not like buying a component with a spec sheet. It is closer to hiring a colleague. Two candidates can have identical credentials and still fit a team completely differently, and you only find out by working with them on the real job.

That reframing changes how a switch should be run. You do not simply swap the endpoint and hope. You budget time to re-tune the prompts, adjust the parameters, and rebuild the small scaffolding that made the previous model reliable. A model that looks worse on day one is often just untuned, and a week of careful adjustment can close most of the distance. The teams that treat a model migration as a shopping decision get burned. The teams that treat it as an integration project tend to land somewhere they are happy with.

It also changes how you evaluate. Benchmarks measure a model in isolation. What matters to a product is behaviour in context: inside your prompts, against your tools, on your users' actual requests. We encourage clients to test candidate models on their own hardest workflows before drawing any conclusion, because the leaderboard cannot see any of the things that decide the outcome.

Where this leaves the open versus closed question

The clear-eyed position, for now, is that this is not a binary. Closed frontier models still earn their place on the most demanding reasoning, and for many teams the reliability is worth the cost and the lock-in. At the same time, the best open models have crossed the line from experiment to viable production option, especially once you invest in tuning them to your workload. A capable open model, properly fitted, can serve as a strong and far cheaper foundation.

We expect the next phase to be less about one universal best model and more about matched pairs: teams choosing the model that fits their particular product, voice, and constraints, the way you would choose a specialist for a specific job. The idea of a single default that every builder reaches for is already fading.

For teams building on top of these models, the practical guidance is steady. Do not chase the leaderboard. Design your system so the underlying model can be swapped with effort rather than agony, because you will swap it. Budget for tuning whenever you move, because the raw model is only the starting point. And judge every candidate on your own work, not on a benchmark someone else designed. The order is unsettled, but the discipline that gets you through it is not.

If your team is weighing a model change or trying to bring the cost of an AI product under control without losing quality, this is exactly the kind of tradeoff we help clients navigate.