When Good Enough Runs Locally
Businesses don't need the smartest model available for every task. Once good enough becomes cheap enough, the interesting question stops being which AI provider to use and becomes which parts of your AI infrastructure you should actually own.
Almost every business using AI today is renting it from someone else’s computer.
That was the correct decision. It might not stay the correct decision.
I’ve been watching the gap between “the best model available” and “a model that is entirely adequate for this task” get narrower, and I think it changes more than people realise.
The Cloud-First Default
Right now, AI in most businesses means an API key.
You sign up with OpenAI, Anthropic or Google, you put the key in an environment variable, and your application sends text over the internet to a data centre somewhere and gets text back.
This made complete sense. Frontier models required infrastructure that no normal business could justify, and the open models you could run yourself were dramatically worse. Not slightly worse. Worse in ways that made them useless for real work.
So the decision was easy, because there wasn’t really a decision.
The Good Enough Threshold
Here’s the thing that has quietly changed.
Most of what businesses actually do with AI is not hard.
Summarising a document. Pulling structured fields out of an invoice. Classifying a support ticket. Searching internal knowledge and answering from it. Rewriting a paragraph. Writing routine code that follows an obvious pattern. Drafting a first response to a customer.
None of that needs the smartest model in the world. It needs a model that is reliably competent, and a lot of models are now reliably competent at exactly this kind of work.
The frontier is still meaningfully ahead. It just isn’t ahead in ways most businesses encounter on a Tuesday afternoon.
When the best model costs several times more than the adequate one, and the difference only shows up on tasks you rarely run, businesses will drift to the adequate one. Not out of principle, out of accounting.
That drift is already happening inside the cloud providers themselves. Half the practical advice in AI engineering right now is some version of “use the cheaper model for this bit”. Once you’ve accepted that argument, you’ve accepted the premise that matters. You’re no longer buying the best available intelligence, you’re buying enough of it.
And once you’re buying enough of it, where it runs becomes a genuine question.
Why This Changes The Economics
Several things are moving in the same direction at once.
Hardware keeps getting cheaper per unit of compute. Memory capacity keeps climbing, and memory is usually the binding constraint locally, not raw speed. Quantisation has gone from a lossy hack to a standard part of the pipeline. Inference software has improved enormously, so the same weights on the same silicon run considerably faster than a year ago. And open-weight models keep closing ground on the closed ones.
But the part I find most interesting is an asymmetry that doesn’t get discussed much.
A cloud model’s price is set by a vendor. It can go down, and it often has, but it goes down when they decide it goes down, and it can go back up when the subsidised land-grab phase ends.
An open model’s cost is set by physics and hardware. Once the weights exist, every improvement in hardware and inference software makes that same model cheaper to run, forever, without anyone’s permission and without a pricing announcement.
Those two curves don’t have to cross for this to matter. They only have to get close enough that the other factors start deciding.
Training Is Not Inference
This is the distinction that gets collapsed most often, and collapsing it leads people to the wrong conclusion.
Training a frontier model is an enormous industrial undertaking. That is not moving into anyone’s server room. If anything it concentrates further, because the capital requirements keep rising.
Inference is a completely different problem. It’s a much smaller computation, run many more times, usually against data that belongs to someone specific.
There is no contradiction in a world with a handful of gigantic training facilities and millions of small inference deployments. That’s arguably the natural shape of it. The expensive, centralised thing produces an artefact, and the artefact runs anywhere. Nobody compiles their own database engine, and nobody sends their queries to the company that wrote it either.
The Part Nobody Argues About
Everything above is a forecast. This bit isn’t.
Right now, a lot of businesses are sending things to overseas providers that they would never email to a stranger. Source code. Client contracts. Medical records. Financial statements. Customer records. Internal documents that describe exactly how the company works.
The contractual protections are usually fine. That’s not the same as the data never leaving the building.
For a lot of organisations, particularly in Australia where data sovereignty comes up constantly in health, legal, government and financial work, this is the whole conversation. I’ve sat in meetings where a genuinely useful project died at the words “so where does the document actually go?”
A local model answers that question in a way no contract can. The document doesn’t go anywhere. That’s it.
Data sovereignty on its own might end up being a bigger driver of local inference than cost ever is.
Rent Versus Ownership
Cloud inference is rent. You pay per unit of use, you commit to nothing, and you scale smoothly from nothing to a lot.
That’s a great deal when usage is low or unpredictable, which describes most businesses in their first year of using AI.
It’s a worse deal when usage becomes steady and high, because the bill scales with success. Every efficiency gain shows up partly as a larger invoice.
Local infrastructure inverts that. Large cost up front, then a marginal cost close to electricity. Terrible if you’re experimenting, quite good if you’ve got a known, constant workload.
Businesses already understand this trade-off. They make it about storage, about networking gear, about whether to lease or buy vehicles. AI compute is unusual right now mostly because it’s new. Give it a few years and “should we own some of this?” becomes an ordinary infrastructure question rather than a philosophical one.
The Answer Is Almost Certainly Hybrid
I’m not predicting everything comes back on-prem. That would repeat the mistake the cloud maximalists made, just in the other direction.
The plausible shape is mixed:
- Routine, high-volume tasks go to small local models
- Sensitive data goes to local models regardless of task difficulty
- Genuinely hard or unusual work goes to frontier cloud models
- Anything you can’t run yet goes to the cloud until you can
Most workloads sit in the first two categories. Most of the value people talk about sits in the third. Both things are true at once.
What That Architecture Looks Like
If you accept the hybrid premise, the design follows fairly directly.
Your internal applications shouldn’t talk to OpenAI. They should talk to one endpoint that you control.
Behind that endpoint sits a router: a few local models, plus optional fallback to whichever cloud providers you use. Requests get routed on the dimensions that actually matter, which are privacy classification, difficulty, latency requirements and cost.
The application doesn’t know or care which model answered. It asked a question and got an answer.
The immediate benefit of this is not saving money, and anyone who sells it that way is overselling it. The immediate benefit is that you stop hard-coding a commercial relationship into fifty different codebases. When pricing changes, when a better open model lands, when legal decides a category of document can’t leave the country any more, you change a routing rule instead of a hundred integrations.
That’s worth doing even if you never run a single model locally.
Why This Might Be An Inflection Point
The current AI build-out assumes that demand for more powerful models keeps growing fast enough to justify historically enormous capital expenditure.
That assumption holds as long as each new frontier model is enough better to pull businesses along with it.
The risk isn’t that progress stops. It’s that progress continues and stops mattering commercially. The new model is better. It’s just not better at extracting fields from invoices, because the old one already did that correctly.
If that happens across enough ordinary workloads while hardware keeps improving, a meaningful share of inference demand quietly stops arriving at the hyperscalers. Not through any dramatic event. Through a lot of individual, boring decisions about what to run where.
Cloud computing emptied a lot of server rooms. AI has odd enough properties, privacy-sensitive inputs, high marginal cost, an artefact that can be copied, that it might partially refill them. Not because the cloud failed, but because this particular workload has a different shape to the ones that moved.
“On-prem AI appliance” could end up as unremarkable as a NAS or a firewall.
Or It Might Not
I’d rather flag the ways I could be wrong than pretend this is settled.
Hardware improvement might not arrive fast enough, or might arrive as expensive accelerators that never reach ordinary workstation budgets.
Open models might stall, or keep lagging in the areas businesses care most about, which is usually reliability and instruction-following rather than headline capability.
Cloud pricing might simply fall far enough to end the argument, and the providers have both the margin and the motive to do that.
And running your own infrastructure has a real cost that cloud pricing quietly absorbs on your behalf. Patching, monitoring, capacity planning, someone’s weekend when it breaks. That’s not free, it’s just not itemised on an invoice.
Frontier cloud models are going to remain important. I use them daily and I’d struggle without them.
The interesting question isn’t whether the crossover happens. It’s where it lands, and for which workloads, and how quickly.
Where I’d Start
If any of this seems plausible, the cheap first move isn’t hardware. It’s the gateway.
Put one internal endpoint in front of your AI usage. Route everything through it. Then measure two things: what data you’re actually sending out, and what each category of request costs you.
Most teams are surprised by both. Once you have those numbers the ownership question stops being speculative, because you can see which workloads are constant, which are sensitive, and which would pay for a box.
You’ve also bought the option to move any of it later without touching application code, which is worth the afternoon on its own.
If you’re weighing up where your inference should run, I’m happy to talk it through and tell you honestly if owning any of it makes sense for you yet.
Because the question is shifting. It used to be “which AI provider should we use?”
Fairly soon it becomes “which parts of this should we actually own?”
John Croucher builds practical software and AI systems for Australian businesses, focused on solving real problems with measurable outcomes.