GPU / AI · 6 min read
Choosing a GPU server for AI inference
The right GPU is determined by the model you serve, the memory it needs, how many requests run at once, and the response time users expect.
Start with VRAM, not the GPU name
Your model, context window, precision, and concurrency determine the memory requirement. Use the VRAM table by model and quantization before you pick a card. For a working cheap 35B example on 2× Tesla P100, see how to run a current 35B cheaply. Leave headroom for the inference runtime, model cache, and traffic spikes. A GPU that only barely fits the model can be difficult to operate in production.
Match the workload
Efficient inference and video tasks may suit a lower-memory GPU. Development, rendering, and data analysis often benefit from a balanced GPU. Large models, scientific computing, and high-concurrency workloads generally need more VRAM and stronger system resources.
Do not ignore the rest of the server
CPU, RAM, NVMe performance, bandwidth, and location still affect the result. Slow data loading, insufficient system memory, or a distant region can make an expensive GPU feel slower than expected.
Questions to answer before requesting a quote
- Which model, quantization, and context length will you run?
- How many concurrent users or requests are expected?
- Is the priority latency, throughput, training, rendering, or cost?
- Do you require a specific region, private networking, or managed setup?