Can an RTX 4090 run Gemma 4 12B IT QAT?
Yes: an RTX 4090 runs Gemma 4 12B IT QAT at 97.7 tok/s at Q4_0 (tested at 8K only), measured on llama.cpp in October 2026.
1 signed run on llm-speed. Single-request decode speed in tokens per second; the short-chat workload unless noted. This run was measured by llm-speed (@llmspeed). Runs measured by llm-speed are credited to @llmspeed.
Measured tok/s by quant and runtime
| Quant | Runtime | Build | Decode tok/s (best run) | Workload | Runs | Source |
|---|---|---|---|---|---|---|
| Q4_0 | llama.cpp | 8931 (9725a313b) | 97.7 | chat-short | 1 | Measured by llm-speed@llmspeed |
Fastest quant that fits: Q4_0 on llama.cpp, 97.7 tok/s (measured by llm-speed; it fits because it ran on this hardware). Quants without a run are not listed.
VRAM: not recorded for these runs. The VRAM calculator gives an estimate (predicted, not measured).
Every measured row
| Run | Date (UTC) | Workload | Quant | Runtime / build | Prompt tokens | Output tokens | Decode tok/s | Prefill tok/s | TTFT ms | VRAM GiB | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|
| r_jfh0qkngw76 | 2026-10-16 | chat-short | Q4_0 | llama.cpp · 8931 (9725a313b) | 124 | 256 | 97.7 | 1109.0 | 112 | – | Measured by llm-speed@llmspeed |
| r_jfh0qkngw76 | 2026-10-16 | chat-long | Q4_0 | llama.cpp · 8931 (9725a313b) | 3191 | 1024 | 86.3 | 1743.0 | 1831 | – | Measured by llm-speed@llmspeed |
| r_jfh0qkngw76 | 2026-10-16 | agent-trace | Q4_0 | llama.cpp · 8931 (9725a313b) | 3939 | 840 | 89.1 | 2305.0 | 983 | – | Measured by llm-speed@llmspeed |
| r_jfh0qkngw76 | 2026-10-16 | concurrent-decode | Q4_0 | llama.cpp · 8931 (9725a313b) | 1024 | 256 | 95.5 | – | – | – | Measured by llm-speed@llmspeed |
Questions
Can an RTX 4090 run Gemma 4 12B IT QAT?
Yes: an RTX 4090 runs Gemma 4 12B IT QAT at 97.7 tok/s at Q4_0 (tested at 8K only), measured on llama.cpp in October 2026.
How many tokens per second does Gemma 4 12B IT QAT get on an RTX 4090?
Decode speed measured by llm-speed, by quantization and runtime: Q4_0 on llama.cpp: 97.7 tok/s. Based on 1 run measured by llm-speed; single-request chat-short unless noted.
What is the fastest quant of Gemma 4 12B IT QAT that fits on an RTX 4090?
Q4_0 on llama.cpp was the fastest measured setting at 97.7 tok/s; it fit because it ran. Other quants are listed only where a run measured them.
How much VRAM does Gemma 4 12B IT QAT use on an RTX 4090?
Memory use was not recorded for these runs. The VRAM calculator gives an estimate, not a measurement.