Hardware FAQ
RTX 4090 — frequently asked questions
Direct answers to the questions local-LLM enthusiasts ask about RTX 4090, drawn from 14 signed runs on llm-speed. Every numerical claim links to a verifiable run permalink at /r/<id>.
What's the fastest LLM on RTX 4090?
The fastest measured LLM on RTX 4090 on llm-speed is gemma3 at 195.0 decode tok/s on ollama 0.31.1 (workload chat-short, run r_dlanfbgym0h). Cite as https://llm-speed.com/r/r_dlanfbgym0h.
This is the headline decode tokens-per-second across every (model, backend) pairing submitted on RTX 4090; faster results may exist on hardware not yet benchmarked, but among signed runs this is the published top.
Can I run a 7B-class model on RTX 4090?
No 7B-class model run on RTX 4090 has been submitted to llm-speed yet, so the canonical "yes/no with a measured tok/s" answer is not currently published.
RTX 4090 has ~24 GB of VRAM/unified memory, which should fit a 7B model at 4-bit quantization; submit a run with "llm-speed bench --models <hf-id>" to populate this answer.
Can I run a 30B-class model on RTX 4090?
No 30B-class model run on RTX 4090 has been submitted to llm-speed yet, so the canonical "yes/no with a measured tok/s" answer is not currently published.
RTX 4090 has ~24 GB of VRAM/unified memory, which should fit a 30B model at 4-bit quantization; submit a run with "llm-speed bench --models <hf-id>" to populate this answer.
Can I run a 70B-class model on RTX 4090?
Not at native precision: RTX 4090 has roughly 24 GB of VRAM/unified memory and a 70B-class model (70-72B parameter range) needs at least 48 GB even at 4-bit quantization, so a 70B-class model won't fit end-to-end on RTX 4090 as a single-GPU/SoC config.
For workloads that do fit, see the RTX 4090 leaderboard for the largest model successfully measured on this hardware.
How does RTX 4090 compare to RTX 5090 for local LLM inference?
For a head-to-head between RTX 4090 and RTX 5090, see the side-by-side comparison page at https://llm-speed.com/vs/rtx-4090-vs-rtx-5090, which lays out every (model, backend) pair where both rigs have a signed run.
As a single-rig anchor, RTX 4090 tops out at 195.0 decode tok/s on gemma3 via ollama (run r_dlanfbgym0h); the RTX 5090 top number is on its own /hw/<slug> page so the comparison stays grounded in measured numbers, not extrapolation.
RTX 4090 vs RTX 5090 (side-by-side) · RTX 5090 leaderboard · RTX 4090 leaderboard
Which backend is fastest on RTX 4090?
The only backend with a signed local run on RTX 4090 so far is ollama, with a top result of 195.0 decode tok/s on gemma3 (run r_dlanfbgym0h); a multi-backend comparison on RTX 4090 is not yet published.
Submit a competing run with "llm-speed bench --backends <other-backend>" on a RTX 4090 machine to populate the comparison.
How much VRAM do I need for local LLM inference on RTX 4090?
RTX 4090 ships with up to 24 GB of VRAM, which sets the ceiling on what fits: a 7B-class model at 4-bit needs roughly 6 GB, a 30B-class model at 4-bit needs roughly 18 GB, and a 70B-class model at 4-bit needs roughly 38 GB.
Long-context workloads add KV-cache pressure on top of the weights — a 32k-token context on a 70B-class model adds another 8-16 GB depending on attention layout, so the practical fit is tighter than the weight-only number.
For exact "did it actually run" evidence on RTX 4090, the leaderboard at /hw/rtx-4090 lists every model that has produced a signed result on this hardware.
Is RTX 4090 worth it for local LLM inference in 2026?
On signed data, RTX 4090 delivers up to 195.0 decode tok/s on gemma3 via ollama (run r_dlanfbgym0h), which puts it comfortably above interactive-coding thresholds (~80 tok/s) for the published top configuration.
"Worth it" depends on your model class: RTX 4090 is most useful for 7B-30B class local models with comfortable headroom.
For "what fits and how fast", the per-model rows on /hw/rtx-4090 are the honest answer; for cross-rig comparisons, see /vs/rtx-4090-vs-rtx-5090.
What quantization should I use on RTX 4090?
On RTX 4090, the top decode rates by quant are Q4_K_M at 195.0 tok/s (gemma3, run r_dlanfbgym0h) and MXFP4 at 141.7 tok/s (gpt-oss, run r_iu2sfa9ykvw); pick the smallest quant whose quality your task tolerates, since on memory-bandwidth-bound hardware decode speed scales roughly inversely with bytes-per-weight.
Quant choice is a quality-vs-speed tradeoff that this hardware FAQ does not arbitrate; llm-speed publishes hardware-side speed, not output quality. For quality scores, see the model card on Hugging Face and the LMSYS Chatbot Arena.
Q4_K_M: 195.0 tok/s (run r_dlanfbgym0h) · MXFP4: 141.7 tok/s (run r_iu2sfa9ykvw)