Why run models locally at all
The cloud APIs are excellent and getting cheaper. So the case for local models is not “cheaper tokens”. It is three other things.
Data stays where it is. Source code, customer tickets, contracts, medical or financial records: for a lot of European teams the question is not whether a provider is trustworthy, but whether the data may leave the building at all. A model on your own hardware turns a compliance discussion into a non-issue.
Cost becomes flat. An agent that reads your repository, runs tests and rewrites files burns tokens by the million. On a metered API that is a variable cost you have to police. On a card you already own it is electricity.
Latency is yours to control. Classification, routing, extraction and “is this ticket urgent” calls are small and frequent. Served locally, they answer in tens of milliseconds and never queue behind someone else’s traffic.
The honest counterweight: the best open-weight models trail the frontier on hard reasoning, and a single GPU has a hard throughput ceiling. Local AI is a tool for a specific class of work, not a replacement for everything.
What a sensible setup looks like
You do not need a cluster. One machine with a modern GPU and 24 to 48 GB of VRAM covers a surprising amount. Ours runs an inference server that exposes an OpenAI-compatible endpoint, so every tool that speaks to OpenAI can be pointed at it by changing one URL.
- One general model in the 27B class for coding agents, drafting and anything that needs judgement. This is the workhorse and it stays loaded.
- One small, fast model (8B or below) for classification, routing and structured extraction, where speed matters more than depth.
- One embedding model for search over documentation, tickets and notes.
- Coding agents on the command line, bound to the local endpoint. We rolled the same agent setup onto a dozen machines in the fleet and proved every one with a real prompt round-trip, not with a config file that looked right.
That is the whole architecture. Everything below is about what happens once real workloads hit it.
What that hardware actually delivers
Numbers first, because “fast enough” is not a number. Our inference box runs two consumer RTX 3090 cards (2 × 24 GB). Measured on 2 September 2026 with the runtime’s own counters, not a stopwatch:
- 27B-class model, 4-bit quantised, held resident with a 262k context window: 52 to 59 tokens per second of generated text on a short prompt. That is faster than most people read, and comfortably fast for an interactive coding agent.
- Prompt processing on the same model: about 940 tokens per second on a 1,600-token prompt. A 20k-token repository excerpt is digested in roughly twenty seconds before the first answer token appears.
- Memory: that resident model occupies 24.5 GB of VRAM, spread across both cards, most of it reserved for the oversized context window (see lesson 02).
The same measurement session produced an unplanned demonstration of lesson 01: when we tried to benchmark a small 8B model next to the resident 27B, the runtime could only find 3 GB of free VRAM for it, spilled the rest to system memory, and the benchmark timed out at four minutes. The big model never noticed. The small one was unusable. Nothing in the logs said “eviction”.
Every lesson on this list was learned by measuring, not by reading a spec sheet. The setup that “should work” and the one that does are rarely the same.
VRAM is one shared budget, and it does not warn you
Load a 27B model next to a couple of small ones and the runtime quietly evicts the small ones to CPU memory when the big one needs the space. Nothing fails. The small models simply drop from hundreds of tokens per second to about 0.2, and every call that depended on them times out ten seconds later. The first time this happened it looked like a network problem.
The fix is boring: decide which model is the warm one, pin it, put a watchdog on the runtime that checks what is actually resident, and schedule big batch jobs so they do not collide with interactive use. Treat VRAM like you would treat a production database connection pool.
Right-size the context window or pay for it on every call
Modern models advertise enormous context windows and the tooling happily configures them by default. We ran one agent at a 262k-token context because that was the number in the model card. Cutting it to 64k, which was still far more than the workload used, made prefill 86 percent faster. The model did not get worse. It simply stopped reserving memory and compute for a window nobody filled.
Set the context per use case. A router needs a few thousand tokens. A coding agent needs tens of thousands. Almost nothing needs the maximum.
Clients do not know the limit, so truncation is silent
Related, and nastier: most client tools do not ask the server what the context limit is. When a conversation grows past it, the prompt is cut from the front, and the model loses exactly the part that carried the system instructions. The agent does not error. It starts behaving as if it had never been told the rules.
Enforce the limit where you can see it, at the gateway or in the agent harness, and log prompt sizes. A chart of tokens per request over time tells you about this problem days before a user does.
Verify at the receiver, never at the sender
A rollout across many machines produces many opportunities to be wrong: an old Node runtime that the agent CLI silently refuses, an environment variable that was set but not exported, a hostname that resolves differently on one host. The only rollout status we trust is a successful prompt and answer on each machine, recorded with the timestamp. Config that “was applied” is an intention, not a state.
Hybrid search beats pure vectors, and it is cheaper
For retrieval over internal documentation we combine classic keyword search (BM25) with embeddings and fuse the two rankings. Compared with feeding whole documents into prompts, this cut the tokens per answer by roughly 60 percent while returning better passages, because exact identifiers, hostnames and error strings are things keyword search finds and embeddings blur.
Know the weak spot: retrieval is good at “where is X described” and poor at “where is X missing”. Absence is not a passage you can rank.
Decide early whether you want a gateway
Once three teams and a dozen tools talk to the same models, you face a choice: one central AI gateway that handles routing, keys, rate limits and logging, or direct endpoints everywhere. Both work. What does not work is drifting into the second and discovering later that nobody can say which tool sent which prompt. Make the decision while it is still cheap.
When local is the wrong tool
Be honest about the boundaries. Reach for a frontier model when the task needs deep multi-step reasoning, when you need hundreds of concurrent users, or when nobody on the team should be operating a GPU box at 2 a.m. The pattern that works best for us is not “local or cloud” but local first, cloud for the hard ten percent, with the routing rule written down and the data-handling boundary explicit for both paths.
A checklist before you start
- Which data may never leave the network? Write it down; it decides the architecture.
- Which three workloads justify the hardware today? Not a vision, three concrete jobs.
- Who owns the box operationally, including updates, monitoring and the 2 a.m. call?
- What is the measured baseline: tokens per second, prefill time, memory per model?
- How will you notice silent truncation, model eviction and drift?
- Where is the line to the cloud, and who may cross it?
Thinking about running models in-house?
We help teams size the hardware, pick models, wire agents and search into existing products, and set up the monitoring that keeps the whole thing honest. Hands-on, with working code at the end.
AI integration services →