Cheapest 70B LLM Rig? Measured Options Compared
Llama-3.3-70B Q4_K_M: two RTX 3090s decoded 47% faster than one 48GB A40 (19.1 vs 13.0 tok/s). 70B-class options compared. Runs linked.
From llm-speed's two Llama-3.3-70B Q4_K_M runs (llama.cpp): two 24GB RTX 3090s decoded about 47% faster than one 48GB A40. The pair decoded 19.1 tok/s (r_n93ukk0mkmm) and the A40 13.0 tok/s (r_19f64yux97o); the tables list it as A40 (45GB), the ~45 GiB of its 48GB that the driver reports as usable. Which setup is cheaper depends on what you pay for it; we don't track prices. Each speed is one run on a different machine and llama.cpp build. Separately, a 96GB M3 Ultra Mac Studio ran Llama-3.3-70B at 16.8 tok/s with MLX and an MLX 4-bit conversion (r_sx3a4y9n-m4). It isn't directly comparable to the two NVIDIA runs: the backend and the 4-bit format differ. To check which Llama 70B quant fits a given GPU, see Llama 3.3 70B on a GPU: VRAM fit.
Measured 70B-class dense rigs: 2x RTX 3090 (24GB) 19.14tok/s on Llama-3.3-70B-Instruct (llama.cpp, Q4_K_M); A40 (45GB) 12.99tok/s on Llama-3.3-70B-Instruct (llama.cpp, Q4_K_M); M3 Ultra (60-core GPU) 16.78tok/s on mlx-community/llama-3.3-70b-Instruct-4bit (mlx, 4-bit). Runtimes and quant formats differ, so these aren't directly comparable. We have no price basis, so this is not a cheapest ranking.
A 70B-class dense model at 4-bit (Q4_K_M / Q4_0 / 4-bit MLX) lives in roughly 40 GB of memory once you add KV cache for a useful context window. That rules out 24 GB single-card consumer GPUs unless you're willing to split layers, and rules in: a pair of 3090s or 4090s (48 GB combined VRAM), an Apple Silicon Max or Ultra with 64 GB+ unified memory, an AMD MI300X or H100 80 GB on the high end, or AMD's Ryzen AI MAX+ 395 with 96 GB. The buy-once cost ranges from ~$1,500 (used dual 3090) to several thousand for an Ultra Mac Studio. Below is every submitted run for a 70B-or-larger model, ranked by decode tok/s; runs on different runtimes and quant formats aren't directly comparable. If your candidate isn't there yet, run the suite and submit — that row will appear next refresh.
Submitted benchmarks
| Hardware | 70B-class model | decode tok/s | MSRP | Run |
|---|---|---|---|---|
| 2x RTX 3090 (24GB) | Llama-3.3-70B-Instruct | 19.14tok/s | — | r_n93ukk0mkmm |
| M3 Ultra (60-core GPU) | mlx-community/llama-3.3-70b-Instruct-4bit | 16.78tok/s | — | r_sx3a4y9n-m4 |
| A40 (45GB) | Llama-3.3-70B-Instruct | 12.99tok/s | — | r_19f64yux97o |
Side-by-side comparisons
Check it for your setup
- Predict your exact setup: estimate whether Llama-3.3-70B-Instruct fits your GPU or Mac at your quant and context, next to measured speeds where we have them.
- Request this pair: ask for a measured Llama-3.3-70B-Instruct run on hardware we have not measured. Requests are ranked by votes.
- Full run table (3 runs): every Llama-3.3-70B-Instruct result we have, with links to each source run.
See also: All hardware · All models · Methodology