Hardware · AI Infrastructure
GPU for AI Guide: Matching the GPU to Your Model and Workload
Choosing a GPU for AI comes down to matching its memory to your model rather than chasing raw compute. VRAM sets the ceiling on what you can run, because the model’s weights, activations, and KV-cache all have to fit in it — a 70-billion-parameter model needs roughly 140 GB at FP16, 70 GB at FP8, or 35 to 40 GB when quantized to 4-bit. For the dominant case of LLM inference, memory bandwidth matters more than TFLOPS, since the GPU reads every weight from VRAM for each token it generates. Training needs three to four times the inference memory because of optimizer states and gradients. In practice, an H100 or H200 is the production standard, with cheaper cards like the L40S or RTX-class GPUs serving smaller models well, and the choice between renting cloud GPUs and owning hardware turns on control, compliance, and volume.
Key takeaways
- VRAM sets the ceiling. Weights, activations, and KV-cache must all fit — size the GPU to the model first.
- Inference is bandwidth-bound. Memory bandwidth, not raw TFLOPS, drives tokens per second during generation.
- Training needs 3–4× the memory. Optimizer states and gradients dwarf the inference footprint; LoRA cuts it dramatically.
- Quantize before adding GPUs. FP16 to FP8 nearly doubles throughput with under 1% quality loss.
- Pick the higher-memory tier when unsure. Running out of memory in production costs more than slight over-provisioning.
Picking a GPU for AI work is where a lot of money gets wasted, usually by buying for headline compute numbers when the real constraint is memory. The right approach starts from your model and workload and works backward to the hardware. This guide explains the specs that actually matter, how to size a GPU to what you’re running, where the current NVIDIA lineup fits, and when to rent versus own.
What actually matters in a GPU for AI?
The instinct is to compare TFLOPS, but for most AI work the first and most important spec is VRAM, the GPU’s memory. VRAM sets a hard ceiling on the model size and context length you can run, because everything the GPU touches during inference has to live in memory at once: the model weights, the activations, and the KV-cache that holds the running context. The weights set the baseline — the model literally has to fit before you can serve it at all — so a GPU with too little VRAM simply can’t run a given model, regardless of how fast it is.
The second spec, and the one most people underrate, is memory bandwidth. The reason it matters is structural: LLM inference during the token-generation phase is memory-bandwidth-bound, not compute-bound, because the GPU reads the entire model from VRAM on every single forward pass to produce each token. At a 70-billion-parameter model in FP8, that’s reading roughly 70 GB per token, so the faster the GPU can read from memory, the more tokens per second it produces. This is also why adding VRAM beyond what your model needs doesn’t increase throughput — at that point you’re bandwidth-limited, not capacity-limited.
Sizing a GPU to your model
Sizing is mostly arithmetic, and getting it right avoids both overspending and the far worse outcome of running out of memory in production. The model weights dominate, and their size depends on the numeric precision you run at. The chart shows how dramatically quantization shrinks a 70-billion-parameter model.
The math behind the chart is simple: multiply the parameter count by the bytes per parameter. FP16 is two bytes per parameter, so a 70B model is about 140 GB; FP8 or INT8 is one byte, halving it to 70 GB; and 4-bit quantization quarters it to roughly 35 to 40 GB. Then add 20 to 30 percent on top for the KV-cache and framework buffers, and remember that context length inflates the KV-cache further. The decision rule that follows is to fit the model in a single GPU with at least 15 percent headroom if you can, and if it doesn’t fit, quantize harder before reaching for a second GPU.
Why training needs far more memory than inference
A common and expensive mistake is sizing a GPU for inference and assuming it will train the same model. Training requires three to four times the VRAM of inference, because it has to hold not just the weights but the optimizer states, the gradients, and the activations from the forward pass. For standard mixed-precision training with the Adam optimizer, the rule of thumb is roughly 16 to 18 bytes per parameter before activations — which means full fine-tuning of even a 7-billion-parameter model needs around 112 to 126 GB of baseline memory, far more than the 14 GB its weights occupy at inference.
This is why parameter-efficient methods have become the default for most teams. LoRA and QLoRA fine-tune only a small set of added parameters, collapsing the memory requirement enormously — as a rough starting point, you can QLoRA-fine-tune a 7B model on 16 GB, a 13B on 24 GB, and a 30B on 48 GB. Full fine-tuning and pre-training of large models still demand multi-GPU clusters with NVLink, but the practical reality for most production work is that LoRA on a single capable card handles the job at a fraction of the cost.
The current NVIDIA lineup
The data-center lineup in 2026 spans a wide range, and knowing where each card fits saves a lot of evaluation time. The table maps the main options to the workloads they suit.
| GPU | VRAM / bandwidth | Best for |
|---|---|---|
| H100 | 80 GB / 3.35 TB/s | Production standard, up to 70B FP8 |
| H200 | 141 GB / 4.8 TB/s | 70B single-GPU, long context |
| B200 | ~180 GB / 8 TB/s | Frontier training, max throughput |
| A100 | 80 GB / 2 TB/s | Research value (no native FP8) |
| L40S / RTX-class | 48–96 GB | Cost-efficient 13–34B inference |
A few distinctions are worth internalising. The H100 remains the production standard thanks to the most mature software ecosystem and wide availability, while the H200 has the same compute but more memory and bandwidth — it’s a memory upgrade, not a compute one, which is exactly what serving 70B models on a single card needs. The B200 brings Blackwell’s FP4 support and several times the inference throughput. One important caveat: the older A100 lacks native FP8 hardware, so it falls back to INT8, and consumer cards like the RTX 5090, while fine for development, carry data-center licensing restrictions under NVIDIA’s EULA that make them unsuitable for production builds.
The software ecosystem matters too
A GPU is only as effective as the software that drives it, and this is where NVIDIA’s dominance comes from as much as the silicon. The CUDA toolkit, paired with mature inference stacks like TensorRT-LLM, vLLM, and Triton, means NVIDIA cards have first-class support across every major framework and quantization toolkit — so a given GPU often delivers far more real-world throughput on a well-optimised stack than its raw specs suggest. The H100’s enduring popularity owes a lot to having the most battle-tested software, not just competitive hardware.
This matters for two practical decisions. First, AMD’s data-center GPUs like the MI300X offer enormous VRAM at competitive prices, and the ROCm software stack has improved steadily with growing PyTorch, vLLM, and SGLang support — but adopting it still means accepting a less mature ecosystem, which is worth it only if its memory advantage solves a real constraint or you’re willing to do the integration work. Second, the inference engine you choose shapes performance as much as the card: most production teams combine TensorRT-LLM or vLLM for serving with techniques like dynamic batching, so factor the software stack into any GPU comparison rather than reading benchmark numbers in isolation.
Why precision and quantization change the math
Quantization — running a model at lower numeric precision — is the single most powerful lever for fitting AI workloads onto affordable hardware, and it’s nearly free in quality terms. Moving from FP16 to FP8 roughly halves the memory footprint and nearly doubles decode throughput, typically with under 1 percent quality loss on most tasks, and Blackwell’s FP4 doubles throughput again for compatible workloads. The terminal captures the memory math that makes this concrete.
# Estimating VRAM for a model (params P in billions) FP16 inference … P x 2 GB (70B -> 140 GB) FP8 / INT8 … P x 1 GB (70B -> 70 GB) 4-bit (Q4) … P x ~0.5 GB (70B -> ~38 GB) + KV-cache … add 20-30% overhead Training (Adam) … P x ~16-18 GB (7B -> ~120 GB) LoRA / QLoRA … base + 10-20% (7B fits on 16 GB) # Quantize before adding a second GPU; FP8 ~ doubles throughput.
There’s a hardware caveat that catches teams out: native FP8 requires NVIDIA’s H-series or newer, because the A-series cards lack FP8 hardware and fall back to INT8, giving up throughput. So if your stack relies on FP8 weights, an A100 isn’t an option no matter how much VRAM it has. The broader principle holds across the board — quantize as far as quality allows before spending on more or bigger GPUs, because the cheapest way to fit a model is almost always to shrink it rather than to grow the hardware.
When do you need multiple GPUs?
The single-versus-multi-GPU decision follows directly from the sizing math. If your model, KV-cache, and overhead fit on one GPU with comfortable headroom, stay single — it gives the lowest latency and the simplest operations, with no inter-GPU communication to coordinate. The moment a model genuinely won’t fit even after quantization, you move to multiple GPUs, but that step isn’t free: tensor parallelism across cards pays an interconnect tax on every forward pass, which is why the quality of the GPU-to-GPU link matters so much.
That link is NVLink, and its bandwidth is why data-center cards command a premium for multi-GPU work. NVLink runs at 900 GB/s on the H100 and H200 and 1.8 TB/s on the B200, versus the much slower PCIe links on workstation cards — and for distributed training across many GPUs, that interconnect bandwidth becomes the bottleneck. Truly large models force this: DeepSeek’s 671-billion-parameter model needs eight H200s just to run. The practical guidance is to quantize and fit on one GPU wherever possible, and only plan multi-GPU and NVLink when the model size genuinely demands it, a topic our LLM hosting guide develops further.
Should you rent cloud GPUs or own the hardware?
Once you know what GPU you need, the deployment question is rent or own, and there’s a useful framing for it: the GPU CAP theorem, which holds that GPU infrastructure can’t simultaneously guarantee control, on-demand availability, and low price — you pick two. Cloud providers, from the hyperscalers to the cheaper neoclouds, give you availability and avoid upfront cost, letting you rent an H100 or H200 on demand, but with less control and weaker compliance guarantees. Owning GPU servers gives you maximum control and the best price at sustained utilisation, at the cost of high upfront spend and longer procurement.
The economics turn on utilisation and the metric that actually matters, cost per token rather than cost per hour. A higher-bandwidth GPU can be cheaper per token despite a higher hourly rate, because it generates more tokens in that hour — an H200 can work out around a third cheaper per token than an H100 even though it rents for more. For steady, predictable workloads at high utilisation, owning hardware usually wins on total cost; for spiky or experimental work, cloud’s flexibility is worth the premium. This is the same own-versus-rent calculus our bare-metal versus cloud guide works through for infrastructure generally.
Is the most powerful GPU always the right choice?
No — and buying the biggest card by default is how budgets get wasted. The right GPU is the one matched to your specific model, precision, and workload, not the one with the highest benchmark numbers. A team running 13B models for internal inference doesn’t need a B200; a cost-efficient card like an L40S or RTX-class GPU delivers most of the capability at a fraction of the price, and over-provisioning compute you can’t use is pure waste. The genuine value tiers exist precisely because most workloads don’t need a frontier flagship.
That said, there’s one direction in which it pays to round up: memory. If your sizing lands between two tiers, lean toward the higher-memory option, because running out of VRAM in production — an out-of-memory crash under load — is far more costly than a slightly over-provisioned GPU, and ECC memory is worth it for long-running production jobs to prevent silent corruption. Match compute to your workload but give yourself memory headroom. For teams that want to own dedicated GPU infrastructure sized to their models rather than renting at a premium, our dedicated servers in Toronto can be configured around exactly the VRAM and bandwidth your workload needs — while the sizing discipline above ensures you buy the right card, not just the most expensive one.