Running Llama or a similar open model on your own infrastructure means no per-token cost and no data leaving your network. It also means you own the hardware sizing, the quality tradeoff, and the operations.
A self-hosted LLM is an open-weight model such as Llama running on hardware you control, usually served through a local runtime. The reasons to do it are data residency, predictable cost at volume, and independence from a vendor. The tradeoffs are capability relative to frontier hosted models, and responsibility for sizing and operations.
No per-token cost once the hardware exists, which changes the economics at volume
No data leaving your network, which is often the actual requirement
Lower capability than frontier hosted models on hard reasoning tasks
Genuinely capable on classification, extraction, summarisation, and routing
THE SETUP GUIDE
The most common mistake is choosing a self-hosted model for a task that needs a frontier model, then concluding that self-hosting does not work. Match the task to the tier.
If the driver is data residency or cost at volume, self-hosting is the right answer. If it is curiosity, use a hosted API until a real constraint appears.
Classification, extraction, routing, and summarisation run well on modest models. Complex multi-step reasoning generally does not. Test on your real data before committing to hardware.
The model must fit in available memory along with its context. Quantisation reduces the requirement at some quality cost. This is the constraint that determines your hardware bill.
Put the model behind an OpenAI-compatible endpoint where possible. Everything downstream then treats it like any other provider and can be swapped without a rewrite.
Route the hard cases to a frontier model and the volume to the local one. Hybrid is almost always better economics than either extreme.
Benchmarks do not predict performance on your documents and your edge cases. Build a small evaluation set from real examples and re-run it whenever you change models.
These are the failures worth designing against before they happen, rather than diagnosing afterwards.
Choosing a small local model for a task that genuinely needs a frontier model
Sizing hardware from the model file size and forgetting context memory
No evaluation set, so quality regressions go unnoticed
Assuming self-hosted is cheaper without counting hardware and maintenance
Coupling application code directly to one runtime instead of a stable API
Ignoring that a GPU sitting idle still costs money
We deploy and integrate self-hosted models where data residency or volume economics justify it, and say so plainly when they do not.
Talk through your setupRequirement analysis and honest build-versus-buy advice
Model selection and quality evaluation on your data
Hardware sizing and provisioning
Runtime deployment behind a stable API
Hybrid routing between local and hosted models
Monitoring, evaluation sets, and ongoing management
At high, steady volume, usually yes. At low or spiky volume, usually no: idle hardware still costs money while an API bills only for use. The crossover depends on your token volume and whether the hardware is already there.
Classification, extraction, routing, summarisation, and drafting are all well within reach of modest open models. Complex multi-step reasoning and long-context synthesis remain noticeably better on frontier hosted models.
It depends on model size and quantisation, and the binding constraint is memory rather than compute. The model plus its context has to fit. Test on your real workload before buying anything.
Yes, and it is usually the best answer. Route high-volume, well-defined work to the local model and escalate hard cases to a hosted frontier model.
These pieces are usually decided together. Getting one right rarely helps if the one underneath it is fragile.
Tell us what you are trying to automate and what you have already tried. You will get a straight answer on the smallest responsible next move, including when the answer is to leave it alone.
Request a systems review