For most of this AI wave, inference has meant a round trip to someone else's data center. That assumption is quietly breaking. Small models — distilled from larger ones and compressed to run in a few gigabytes of memory — now handle a useful share of everyday language work on a laptop, a phone, or an industrial PC bolted to a rack. This briefing covers what moves when the model moves.
Key takeaways
- Distilled, quantized models now handle classification, extraction, and routine drafting on ordinary hardware — no round trip required.
- Local inference keeps data inside your boundary by default, which changes the conversation for regulated and disconnected environments.
- Latency and per-call cost drop toward zero; a capability ceiling and fleet management take their place as the constraints to design around.
- The pattern that works is hybrid: small models local for routine steps, a hosted model for the hard ones, with routing rules you can audit.
What actually changed
Two mechanisms did the work. Distillation trains a small model to reproduce the behavior of a much larger one on a defined slice of tasks — trading breadth for a fraction of the footprint. Quantization stores the model's weights at lower numeric precision, shrinking memory and compute needs again with little accuracy loss on those same tasks. Stacked together, they produce models that fit on hardware you already own.
The honest caveat: these are not frontier models. A distilled model does not plan a complex workflow or reason through a novel exception. What it does well is the high-volume middle of operational work — classify this message, extract these fields, summarize this known document format, flag this record for review — at near-zero marginal cost and without the data ever leaving the machine.
Where local inference lands first
- Field and floor operations. Inspection notes, work orders, and equipment logs processed where connectivity is poor or forbidden — a maintenance bay, a remote site, a vessel. The system works the same offline as on.
- Privacy-default intake. Documents classified, redacted, and structured locally before anything crosses a boundary — a design that supports compliance reviews instead of complicating them.
- High-volume, low-stakes steps. Routing, tagging, and de-duplication that would be uneconomical at hosted-model prices become effectively free — which changes which pipelines are worth building at all.
The question stops being “is our data safe in transit?” and becomes “does it need to travel at all?”
The new constraints
Local inference trades one set of constraints for another. The first is the capability ceiling: a small model must be evaluated against your actual documents and formats before it earns a place in a workflow, because the failure mode is quiet — plausible output that's subtly wrong. The second is fleet discipline: a model running on two hundred devices is a deployment target, with versioning, rollback, and telemetry, not a file someone copies around. Neither constraint is exotic. Both are the kind of engineering that separates a demo from a system.
How this shows up in a tailored system
The designs we ship are hybrid by default: a small local model handles the routine steps where it clears the accuracy bar, and the system escalates to a hosted model — or to a person — when confidence drops or stakes rise. The routing rules are written down, tested during scoping, and auditable after. And the review rule doesn't move: consequential outputs are signed by a named reviewer regardless of where the inference ran. Local pre-processing also feeds our data work — records cleaned and structured at the edge arrive analytics-ready, which shortens every downstream project.
This briefing pairs with our note on agent interoperability standards — portable integrations and portable inference are two halves of the same buying position. For how we handle data boundaries in practice, read trust & security, or see the data & analytics practice.