People treat local vs cloud like a sports rivalry. Treat it like plumbing.
Sometimes you want the water company. Sometimes you want a well on your property. Both can be smart. Both can be dumb if you ignore the job.
Run both when it helps. The decision is not ideology. It is constraints.
When local actually wins
Privacy and data gravity. If the text cannot leave the building, local is not a preference. It is the door. Think internal notes, regulated content, or anything your counsel already has feelings about.
Cost at steady volume. Cloud tokens feel cheap until an agent loops, a batch job wakes up, or the whole team “just tries something” all day. A box you already own can make heavy draft work boringly affordable.
Latency on the LAN. Some internal tools feel snappy when inference is next to the app. Round trips add up when you chain tool calls.
Offline and control. Control matters: pin a model version, survive vendor outages, avoid surprise deprecations the week before a launch.
Customization without shipping secrets. Fine-tunes and specialized rerankers on proprietary text are easier to reason about when the weights sit where your security team can point at them.
When cloud still wins (often)
Best-in-class quality for hard judgment. For messy reasoning, coding help, and delicate customer wording, the frontier cloud models still pull ahead more days than not. Local is catching up. “Catching up” is not the same as “caught.”
Speed of iteration. Need a new multimodal model tomorrow? Cloud. Need to A/B two vendors this afternoon? Cloud. Local ops is a real job: drivers, VRAM, quantization tradeoffs, queueing.
Burst scale. Launch day traffic and one-off research spikes are easier when someone else owns the GPUs.
Ecosystem glue. Tooling, eval harnesses, and enterprise contracts sometimes assume an API. Fighting that for purity can cost more than the tokens.
A practical split
A boring pattern works well:
- Cheap or local models for drafts, classification, embedding, and first-pass summarization
- Stronger cloud models for final judgment, tricky coding, and anything customer-facing where tone and correctness matter
- Clear rules for what data is allowed where
That is model routing with a seatbelt. It keeps the bill down without forcing every task through a blender on your desk.
The hidden costs people skip
Local is not free because the model weights are downloadable.
You pay in:
- hardware and electricity
- somebody’s time keeping things alive
- quality gaps that create silent errors
- slower experimentation
Cloud is not free because the invoice is itemized.
You pay in:
- data handling agreements
- rate limits at the wrong moment
- vendor changes
- team habits that treat tokens like air
Write both cost curves down before you pick a side.
A decision cheat sheet
Ask:
- Can this data leave our environment?
- Is quality delta visible to a customer or only to an internal draft?
- Is volume steady or spiky?
- Do we have a human who can own the box?
- What is the failure mode if the model is a little worse?
If (1) is no, local or a private deployment path. If (2) is customer-visible and (5) is scary, bias to higher quality even if it costs more. If volume is huge and tasks are narrow, local or small models shine.
Hybrid is not a cop-out
Hybrid is how adults run systems.
Embed locally, generate remotely. Draft locally, rewrite remotely. Run an on-prem gateway that enforces allowlists while still calling a cloud model for the hard turn. Use cloud for building, local for a constrained production path once the prompt stabilizes.
Purity is a blog comment. Production is a diagram with arrows and a budget.
What to tell teams arguing about it
Stop asking “are we a local shop or a cloud shop?”
Ask “what is the unit of work, what is the risk, and who gets paged?”
Then pick the simplest architecture that meets the bar. Simple architectures ship. Clever ones become identity projects.
Bottom line
Local models beat cloud APIs when control, privacy, and steady compute matter more than peak smarts.
Cloud APIs beat local when quality, burst, and iteration speed matter more than owning the stack.
Use both in the same week without writing a manifesto. The manifesto does not page you. The outage does.
