GPU / AI · 24 Sep 2026 · 12 min read

How to run a current 35B model cheaply (2× P100 + 2080 Ti MTP)

You do not need an H100 to serve a modern 35B-class model for personal or small-team use. A used 2× Tesla P100 target plus a GTX 2080 Ti as the MTP / draft GPU is one way to get interactive speed without hyperscaler hourly rates. Below is the architecture, the numbers from my stack, and the limits so you do not copy it blindly.

The measured setup

These figures are from my own machine, not a rented lab and not a vendor slide. Treat them as one working configuration: Qwen3.5 35B, quantized inference, interactive decode (tokens generated after the prompt is processed).

Live measurement from this stack (24 Sep 2026)

llama.cpp on the local box currently serves Qwen3.6-35B-A3B Q4_K_M (file Qwen3.6-35B-A3B-Q4_K_M.gguf, ~34.7B params, ~20.4 GB, MoE with a small active path). Timed completions (batch 1, MTP off) returned:

• Decode: 44.8 tokens/s (128 tokens) and a repeat at 44.9 tokens/s (192 tokens)
• Prefill: 31.0 tok/s on a 17-token prompt, 53.6 tok/s on a 20-token prompt (both too short to treat as RAG numbers)
• MTP / speculative decoding: off (speculative: false on the running server; 2080 Ti stays idle)

That ~45 tok/s is in the same band as the ~40 tok/s figure for 2× P100 without MTP. It is not a dense 35B; A3B MoE is why Pascal can feel this fast. Qwen3.8 27B at ~20 tok/s was a separate run without MTP. There is no 27B GGUF on this host right now, and the start script leaves MTP off because a 2080 Ti draft has aborted with CUDA invalid argument. Do not restart the live server to force MTP until you accept that crash risk.

What it costs (used market, Sep 2026)

These are asking / sold ranges, not a quote. Check eBay, Newegg refurbished, and server-parts shops before you buy. Tesla P100s are passive (no fan) and have no display ports — they need server airflow or a blower/shroud, plus the right power connector.

A pair of P100s for about the price of one used 2080 Ti is why this build exists. You are paying for 32 GB of HBM2, not for modern tensor cores.

Models that run on 2× P100 (32 GB)

Fit is about quantized weights + KV cache, not the marketing name. Numbers in bold are measured on this machine. Other rows are planning guidance — confirm with your quant and context. See also the VRAM table.

Rule of thumb: Q4 27B–35B chat is the sweet spot for dual P100. If you need 70B or long 32k RAG, spend on VRAM, not on another Pascal card.

Model “price”: weights are free, APIs are not

Qwen3.5 35B and Qwen3.8 27B are open weights. You do not pay per download. You pay the GPUs (~$190–$360 for two P100s), power, and your time. The money you save is against API bills and rented datacenter GPUs, not against a Qwen license fee.

API rates change. Confirm the vendor’s price page before you treat the table as accounting.

How much you can save (worked examples)

Assume GPU pack ~$500, amortized over 24 months (~$21/month), plus power. Measured GPU decode on this 35B run was about 196 W across three cards (not the 750 W nameplate). At $0.12/kWh: 196 W × 8 hours/day × 30 days ≈ $5.70/month for GPU silicon during active generation. GPU idle on the same host was about 97 W (~$8.40/month if left loaded in VRAM 24/7). Wall power is higher (CPU, PSU loss). Total ballpark ~$30–$45/month to own the stack if you actually run it daily.

• Versus always-on H100: ~$2,500+/month vs ~$40. Savings on the order of $2,000+/month — but you also lose H100 speed, VRAM, and SLAs. Fair only if a 35B P100 box meets the job.
• Versus GPT-4o at 10 million input + 10 million output tokens/month: ~$25 + $100 = ~$125/month API vs ~$40 hardware. Save about $80/month after the GPUs are paid off; first two months the hardware still wins vs that API volume. Privacy is extra.
• Versus GPT-4o mini at the same 20 million tokens: ~$1.50 + $6 = ~$7.50/month. The P100 box can lose on cash if you only need a mini-class model. You still win on air-gap and on matching a 35B locally.
• Versus Qwen flash API: similar story to mini — hosted flash is cheap. Self-host if you want the 35B locally or zero vendor telemetry, not if you only wanted flash quality.

Break-even vs flagship APIs is “heavy personal or small-team use.” Break-even vs flash/mini APIs is mostly privacy and capability, not rupees, unless volume is huge.

Which model for which work

Default: 35B + MTP for quality and speed, 27B without MTP if you want simpler ops and more VRAM headroom. Re-test 27B with MTP before you decide it is “the slow one.”

Qwen3.8 27B + 2080 Ti MTP — not measured on this host

The ~20 tok/s figure is 2× P100, no MTP. This box has no Qwen3.8 27B GGUF on disk. The live server is 35B without a draft model; the start script comments that a 2080 Ti draft aborts with CUDA invalid argument. Those two cells stay empty until a 27B file is loaded and a draft session actually stays up.

Power: spec vs nvidia-smi

NVIDIA’s board power cap is not what you pay at the wall. Nameplate TDP is still useful for PSU sizing:

• Tesla P100 16 GB PCIe: 250 W cap each (500 W for the pair if both sit at the limit).
• GeForce 2080 Ti: 250 W TDP (260 W reported as power.limit here).
• Three-card worst case on paper: ~750 W of GPU silicon, plus CPU, RAM, and PSU inefficiency — plan a 1000 W+ quality PSU and real airflow. Passive P100s will thermal-throttle in a desktop with no front-to-back blast.

Live capture during Qwen3.6-35B-A3B Q4_K_M decode (MTP off, 24 Sep 2026). The 2080 Ti is idle; both P100s are in llama-server. Same run decoded at 44.9 tok/s.

A second sample used the CSV query (power.draw,utilization.gpu,memory.used). Watts move around during decode; both samples stay far under the 750 W nameplate:

Temps during that sample were 35 / 34 / 38 °C. This is board power.draw, not wall watts. MTP-on and 27B rows are still missing because those runs were not taken. Do not size a UPS from the 250 W sticker alone.

Why “old” GPUs can still run a latest 35B

Three things changed more than the silicon on a P100:

1. Quantization. Q4/Q5/Q8 shrinks weights so 35B fits in ~20–28 GB plus cache instead of ~70 GB FP16. See the VRAM planning table.
2. Tensor parallel. Two 16 GB P100s present ~32 GB to a framework that can split layers or columns across cards. One P100 cannot hold this model comfortably.
3. Speculative decoding / MTP. The big model does not have to emit every token the hard way. A smaller, faster GPU proposes a short burst of tokens; the 35B model checks them in a batch. High acceptance → more tokens per second. Low acceptance → you pay verify cost and fall back toward the 40 tok/s baseline.

Pascal (P100, 2016) has excellent HBM2 bandwidth for its era and poor support for the tricks that make an H100 cheap per token at the datacenter. You are buying used VRAM and bandwidth, not FP8 tensor cores. That trade is rational for a 24/7 personal 35B server. It is irrational for training a 70B model.

What “2080 Ti as MTP” means in practice

MTP here is multi-token prediction / speculative decoding. The 2080 Ti (Turing, 11 GB) is a poor home for a 35B target, but it is a strong drafter: high clocks, enough memory for a small draft head or a distilled draft model, and it does not steal HBM from the P100s.

• The target (2× P100) holds Qwen3.5 35B and is the source of truth for every accepted token.
• The draft / MTP worker (2080 Ti) proposes several tokens ahead.
• The target verifies a chunk at once. Accepted tokens skip extra sequential steps. That is the jump from ~40 to ~55 tok/s.

If the draft is too dumb, acceptance collapses and 55 tok/s will not show up. Tune draft length; do not assume more speculation is always faster. Pair a draft that is in-family with Qwen when you can.

How to think about the benchmark

Tokens/second without context is how vendors lie to themselves. When you copy this build, write down:

• Prefill vs decode. Quote decode for chat UX; quote prefill if your app is RAG with huge prompts.
• Batch size. These numbers are for interactive use (small batch). Many concurrent users can raise throughput and destroy latency.
• Quant and context. Longer context grows KV cache on the 32 GB P100 pool first. That can force a lower context cap or offload, which drops tok/s.
• PCIe, not NVLink. Two P100s in a workstation often talk over PCIe. Tensor parallel pays a tax. A dual-P100 SXM board with NVLink (if you can find one) behaves better than two blower cards in x8/x8 slots.

Relative GPU scores on the benchmarks page compare rental classes (L4, RTX, A100). They will not list a dual-P100 hobby build. Use this article for the used-Pascal path; use the finder when you want a supported GPU server from a provider.

Bill of materials (the cheap part)

• Two used Tesla P100 16 GB (~$95–$180 each in Sep 2026 listings).
• One used 2080 Ti for MTP (~$220–$255 typical).
• A board with enough PCIe, a 1000 W-class PSU, 64 GB+ host RAM, NVMe for weights.

GPU silicon around $400–$620 plus a host you may already own. A month of always-on H100 cloud still costs more than this pile if you actually leave the 35B running. Electricity is the ongoing bill: this 35B decode sat near 196 W of GPU, not 400–600 W. Wall draw is still higher; Indian commercial tariffs make 24/7 use real money even at that load.

When this is the right answer

• You want a private 35B chat / coding / RAG box on at home or in a colo, running all month.
• You accept quantization and a moderate context window.
• You can maintain drivers, CUDA, and a serving stack (llama.cpp, vLLM, SGLang, or similar with speculative decoding).
• ~40–55 tok/s feels interactive enough (it does for most chat).

When you should not copy it

• Training or heavy LoRA on 35B+ — buy or rent Ampere/Hopper. Read choosing a GPU for inference if your job is actually serving, not training.
• 70B+ at decent context — 32 GB combined is the wrong ceiling. Step to 48–80 GB class cards on GPU servers for AI.
• Customer SLA, compliance, or zero-ops — a used Pascal pair is not a provider GPU server.
• Maximum tok/s per watt — a single modern 24–32 GB RTX will often beat this on efficiency even if it costs more to buy.

A practical build order

1. Confirm the 35B quant + context you will ship using the VRAM table. If weights + cache exceed ~26–28 GB, two P100s will OOM or thrash.
2. Get decode working on 2× P100 only. Record tok/s on a fixed prompt (for example 512-token prompt, 128 generated, batch 1). That is your 40 tok/s class baseline.
3. Add the 2080 Ti as draft/MTP. Sweep draft length until acceptance is high and decode rises. If speed drops, the draft is hurting you.
4. Measure prefill on a 2k and 8k prompt so RAG users are not surprised.
5. Only then decide whether a rented 24 GB RTX or 80 GB A100 from a listed GPU offer is actually cheaper for your hours-per-month.

Cost logic in one paragraph

Cloud H100 is priced for burst training and dense production. A used dual-P100 box is priced for “always on, good enough 35B.” If you generate millions of tokens per day, tok/s and watts dominate. If you chat a few hours a day, the purchase price of used cards dominates. MTP is the cheap speedup: one extra 11 GB Turing card bought the ~15 tok/s that would otherwise mean a newer target GPU.

Need a supported GPU instead of a used-Pascal project? Compare GPU servers Find My Server