VRAM is the spec · Chapter 04

Local AI picks.

For local AI one number decides everything: VRAM. The model either fits in memory or it does not, and no amount of GPU speed rescues a model that does not fit. Rule of thumb at 4-bit quantization: 8GB runs 7B-8B models, 12GB runs 13B-14B, 16GB runs the low 20Bs, 24GB runs 32B-class, and 32GB runs 32B with room to breathe. CUDA support is the second spec: NVIDIA cards run everything, AMD cards run llama.cpp and Ollama well but lock you out of CUDA-only tools.

The VRAM ladder

What fits where

Approximate memory needed to run popular open-model sizes at 4-bit quantization. Leave headroom for context; long conversations eat memory too.

Model class	Needs (Q4)	Cheapest way to run it
7B-8B	~5GB	Any 8GB card runs these; a used RTX 3060 12GB runs them with room to spare
13B-14B	~9GB	Used RTX 3060 12GB is the budget ticket; any 12GB card works
22B-24B	~14GB	16GB cards: the RTX 5060 Ti 16GB is the cheapest new option with CUDA
32B	~20GB	24GB territory: used RTX 3090 for value, RTX 4090 for speed, RX 7900 XTX for the AMD route
70B	~40GB	Beyond any single consumer card at Q4: two RTX 3090s over NVLink, an RTX 5090 with partial CPU offload, or rent the compute