GPU / AI · 4 Oct 2026 · 14 min read
Self-host LLM vs API: when a dedicated GPU actually wins
Teams keep asking the same question after their first big API invoice: should we run the model ourselves? The honest answer is not “always” or “never.” It is a utilization and quality problem — plus privacy. This guide gives you a decision framework you can actually use before you buy a GPU server.
Two different cost models
An API bills by tokens. When traffic is zero, your bill is zero. When traffic spikes, cost scales with you. That is a structural advantage for early products, demos, and bursty internal tools.
Self-hosting (owned or rented dedicated GPU) bills by time the machine exists. Whether you serve one request or a full batch queue, the box costs roughly the same that hour. Tokens get cheap only when you keep the GPU busy. Idle GPUs are how “self-hosting is 10× cheaper” posts become expensive lessons.
| Model | You pay for | Wins when | Loses when |
|---|---|---|---|
| API (pay per token) | Input + output tokens, sometimes cached prompts | Low volume, spikes, uncertain demand, frontier quality needed | Steady high volume against premium model prices; strict data rules |
| Self-host (dedicated GPU) | GPU rent / ownership + power + ops time | Steady load, open weights good enough, privacy / latency control | GPU sits idle; nobody can run inference ops; cheap open-model APIs undercut you |
Important nuance: a hosted API of the same open weights (someone else’s Llama/Qwen endpoint) is often cheaper than your single-tenant GPU, because they keep batches full across many customers. Self-hosting wins more often against premium frontier APIs, or when control matters more than the last cent.
The only math that matters: utilization
Pick a realistic monthly GPU cost for the class of card you need (entry used cards for small models, modern data-center GPUs for 70B-class serving). Divide by how many useful tokens you actually produce. That is your true cost per million tokens — not the theoretical peak if the card never sleeps.
| Average GPU utilization | What it usually means | Cost outcome vs API |
|---|---|---|
| ~10–20% | Chatty demos, night-time idle, spiky product traffic | API almost always cheaper for the same quality tier |
| ~40–60% | Real product baseline with some quiet hours | Can beat expensive frontier APIs; still loses to cheap open-model APIs |
| ~70%+ | Batch jobs, always-on agents, high QPS APIs | Self-host economics get serious — if ops cost is counted honestly |
There is no universal “X million tokens and you break even” number. Break-even moves with: GPU monthly price, model size, quantization, batching, prompt/response lengths, and which API price you compare against. Do the spreadsheet with your last 30 days of token usage, not a blogger’s H100 fantasy.
What “self-host” actually includes
People underestimate the bill of materials. Hardware rent is only the first line.
- GPU + host — VRAM first (see how much VRAM you need), then CPU RAM for KV cache / offload, then fast local disk for weights and swaps (NVMe vs SSD).
- Inference stack — vLLM, TensorRT-LLM, llama.cpp, Ollama, TGI, or similar. Someone must upgrade drivers, CUDA, and model files when security and quality move.
- Reliability — one GPU is not HA. Production often means two replicas so a deploy is not an outage.
- Observability — token latency, queue depth, OOM kills, disk fill from Hugging Face caches.
- People time — even half a day a week of GPU ops can erase “savings” for a small team.
If nobody on the team can keep a Linux box patched and a model serving process healthy, stay on an API or buy managed inference — not a bare GPU you will fear to touch.
When self-hosting is the right call (even before perfect math)
- Data cannot leave your perimeter — legal, health, finance, customer PII, or internal codebases. Cost is secondary to control.
- You already burn a large, steady API budget — think “painful every month,” not “interesting experiment.” Then model a dedicated GPU against that invoice.
- Open weights meet quality for most traffic — support bots, RAG over your docs, classification, extraction, code assist on known stacks. Frontier models for the hard 10–20% only.
- Latency or region matters — keep inference next to your app in the same data center or country.
- You can fill batches — background jobs, overnight processing, and multi-tenant internal tools pack the GPU better than one interactive chat user.
When the API should win
- You are still finding product-market fit; token volume is unpredictable.
- You need the latest frontier model quality and no open weight is close enough.
- Traffic is bursty (launches, business hours only, weekend silence).
- Your team has zero GPU operations capacity and no budget for managed help.
- A cheap open-model API already undercuts what you would pay for a lonely dedicated card.
For most day-to-day coding and writing, start with a low-cost API such as DeepSeek V4 Flash (deepseek-flash) before you rent a GPU. Flash often beats chat subscriptions and premium frontier defaults on unit cost — then revisit this page when volume or privacy forces the hardware question.
A practical decision order
- Export last month’s token counts and dollars by model/provider.
- Split traffic: which requests need frontier quality vs “good enough open weights”?
- Size VRAM for the open model you would actually run (Q4/Q5/Q8, context length, concurrency) — start with GPU inference buying and the VRAM tables.
- Get a real monthly quote for that GPU class on a dedicated GPU server (not a hyperscaler on-demand sticker alone).
- Assume 40–60% utilization unless you have proof of higher. Recalculate cost per million tokens.
- Add ops hours. If the spreadsheet still wins and privacy/latency help, self-host the bulk path.
- Keep a frontier API route for the hard tail. Hybrid is the default adult architecture in 2026.
Hardware tiers that match real workloads
Match the job, not the marketing name on the GPU.
| Workload | Typical approach | Notes |
|---|---|---|
| Small / mid open models, experiments, private RAG | Used or mid-tier consumer/data-center cards; sometimes multi-GPU Pascal/Turing builds | See the hands-on cheap 35B on 2× P100 path for budget local serving — not for every production SLA. |
| Production 7B–34B APIs with concurrency | 24–48 GB class GPUs, strong host RAM, NVMe for weights | Prefer modern inference engines with continuous batching over toy single-stream setups. |
| 70B-class / heavy context / many users | High-VRAM cards, sometimes multi-GPU with fast interconnect | Re-check economics carefully; open-model APIs and reserved cloud GPUs compete hard here. |
Bare metal still matters for steady inference: predictable latency, full VRAM, no noisy neighbors. That is the same reason teams revisit dedicated hardware for AI — see bare metal vs cloud when you are choosing the host, not only the model.
Hybrid routing: the pattern that usually wins
Pure ideology loses money. A common three-tier split:
- ~70–80% of requests — cheap bulk path: DeepSeek Flash API and/or a self-hosted open model once utilization justifies it (summaries, RAG, structured extraction, internal chat).
- ~20–30% hard cases — frontier API or DeepSeek Pro (complex reasoning, tool use that fails on Flash, rare languages/domains).
- Spikes — overflow to API instead of buying a second GPU you will idle all month.
Implement routing in your app layer: confidence scores, task tags, or simple rules (“code review → frontier,” “ticket triage → Flash/local”). Measure quality and cost weekly; adjust the split.
Honest caveats
- Model and API prices move monthly. Recalculate; do not tattoo a 2025 blog number onto a 2026 budget.
- Self-hosting does not remove compliance work — you now own logs, access control, and patching.
- “We have a GPU” is not a strategy. Throughput, queueing, and evals are the product.
- Training from scratch is a different problem. This article is about inference serving economics.
Short answers
- Low / spiky traffic: API — often DeepSeek Flash for day-to-day work.
- Steady volume + good open weights + ops skill: dedicated GPU for the bulk path.
- Privacy or residency requirements: self-host even if the spreadsheet is close.
- Default architecture: hybrid — Flash/local for volume, frontier API for the hard tail.
- Next step: size VRAM, then compare real GPU monthly quotes — not vibes.
Compare GPU servers DeepSeek Flash day-to-day guide VRAM planning guide Find My Server