Can an RTX 5090 run Gemma 4 12B IT QAT?

Yes: an RTX 5090 runs Gemma 4 12B IT QAT at 141.1 tok/s at Q4_0, measured on llama.cpp with 16K context in September 2026.

9 signed runs on llm-speed. Single-request decode speed in tokens per second; the short-chat workload unless noted. 6 of 9 runs were measured by llm-speed (@llmspeed); the other 3 are community-submitted (signed on the submitter's own machine, not measured by llm-speed, and not used for llm-speed verdicts). Runs measured by llm-speed are credited to @llmspeed. Verdicts and headline numbers use only llm-speed's own runs and fit tests.

Measured tok/s by quant and runtime

Quant	Runtime	Build	Decode tok/s (best run)	Workload	Runs	Source
Q4_0	llama.cpp	1 (9725a31)	141.1median of 3 runs: 139.5	chat-short	3	Measured by llm-speed@llmspeed
Q4_0	llama.cpp	8931 (9725a313b)	130.6median of 3 runs: 130.2	long-context-decay	3	Measured by llm-speed@llmspeed
Q4_0	llama.cpp	1 (9725a31)	142.6median of 3 runs: 136.3	long-context-decay	3	Community-submitted

Fastest quant that fits: Q4_0 on llama.cpp, 141.1 tok/s (measured by llm-speed; it fits because it ran on this hardware). Quants without a run are not listed.

VRAM used: 7.75 GiB runtime-reported GPU allocation for model, context and compute buffers with a 16,384-token context configured (r_v5arqmazf31; a runtime-log allocation, not peak device memory).

Every measured row

Run	Date (UTC)	Workload	Quant	Runtime / build	Prompt tokens	Output tokens	Decode tok/s	Prefill tok/s	TTFT ms	VRAM GiB	Source
r_v5arqmazf31	2026-09-05	chat-short	Q4_0	llama.cpp · 1 (9725a31)	121	256	141.1	2027.9	60	7.75	Measured by llm-speed@llmspeed
r_c2podhe9qwc	2026-09-05	chat-short	Q4_0	llama.cpp · 1 (9725a31)	121	256	139.5	1958.9	62	7.75	Measured by llm-speed@llmspeed
r_8xmm65n1fhq	2026-09-05	chat-short	Q4_0	llama.cpp · 1 (9725a31)	121	256	137.3	2316.4	52	7.75	Measured by llm-speed@llmspeed
r_c2podhe9qwc	2026-09-05	chat-long	Q4_0	llama.cpp · 1 (9725a31)	3188	620	137.1	5118.0	623	7.75	Measured by llm-speed@llmspeed
r_v5arqmazf31	2026-09-05	chat-long	Q4_0	llama.cpp · 1 (9725a31)	3188	620	135.6	5147.7	619	7.75	Measured by llm-speed@llmspeed
r_8xmm65n1fhq	2026-09-05	chat-long	Q4_0	llama.cpp · 1 (9725a31)	3188	620	135.6	5523.3	577	7.75	Measured by llm-speed@llmspeed
r_-lh-vczf7ik	2026-09-06	long-context-decay	Q4_0	llama.cpp · 1 (9725a31)	23872	98	142.6	7115.8	3355	–	Community-submitted
r_1srqmpjnkgd	2026-09-06	long-context-decay	Q4_0	llama.cpp · 1 (9725a31)	23872	98	136.3	7161.0	3334	–	Community-submitted
r_8q-uf-rq-gp	2026-09-06	long-context-decay	Q4_0	llama.cpp · 1 (9725a31)	23872	98	134.2	6022.4	3964	–	Community-submitted
r_9suyze4n7lx	2026-10-22	long-context-decay	Q4_0	llama.cpp · 8931 (9725a313b)	23872	98	130.6	5881.5	4059	–	Measured by llm-speed@llmspeed
r_fzn_lluvw14	2026-10-22	long-context-decay	Q4_0	llama.cpp · 8931 (9725a313b)	23872	98	130.2	5891.1	4052	–	Measured by llm-speed@llmspeed
r_kdmexe0lkqn	2026-10-22	long-context-decay	Q4_0	llama.cpp · 8931 (9725a313b)	23872	98	129.8	5965.8	4001	–	Measured by llm-speed@llmspeed

Questions