Skip to content
llm-speed

Gemma 4 12B IT QAT

16 workload results across 2 hardware configurations.

Model weights used in these runs:

Can Gemma 4 12B IT QAT run on your hardware?

Fastest decode speed on each GPU or Mac, from llm-speed's own runs and community-submitted runs. Each page lists every quant and runtime and labels which runs llm-speed measured.

For a standard Q4_0 quantization using llama.cpp, here are measured results for short-chat generation.

On llama.cpp Q4_0, gemma-4-12b-it-qat is fastest on the RTX 5090 at 139.5 tok/s, 43% faster than the RTX 4090 (97.72 tok/s).

CardDecode tok/sRunSource
RTX 5090139.5 (median of 3 runs)r_c2podhe9qwc (closest to median)llm-speed (@llmspeed)
RTX 409097.72 (single run)r_jfh0qkngw76llm-speed (@llmspeed)

gemma-4-12b-it-qat, Q4_0, llama.cpp, chat-short, batch 1. Only cards with their standard memory are listed; CPU-only hosts are excluded. Runs use different llama.cpp versions (RTX 5090: llama.cpp commit 9725a31; RTX 4090: 8931 (9725a313b)), so treat this as measured reference points, not a controlled ranking. Community-submitted runs are signed on the submitter's own machine and are not measured by llm-speed; llm-speed verdicts use only its own runs.

Original test · 5 September 2026 · three repetitions

How fast is Gemma 4 12B IT QAT on an RTX 5090?

Measured by llm-speed (@llmspeed): 139.5 tokens per second (median of three runs; best 141.1) in short chat with Q4_0 quantization, using one RTX 5090 with 32 GB VRAM and llama.cpp. Median time to the first token was 59.7 ms with the model already loaded. These are single-user, thinking-off results.

Median of three runs; parentheses show the observed minimum–maximum, not a confidence interval.
WorkloadDecode tok/sFirst token (ms)Prompt tokensOutput tokens
chat-short139.5 (137.3–141.1)59.7 (52.2–61.8)121256
chat-long135.6 (135.6–137.1)619.3 (577.2–622.9)3,188620

Does it fit in 32 GB of VRAM?

Yes, this tested configuration offloaded all 49/49 layers to the GPU. The runtime reported 7.75 GiB (7,931 MiB) of GPU allocation for model, context and compute buffers combined. The model file is 6.98 GB on disk.

Allocation is from the runtime log, not a measurement of peak whole-device memory. A 16,384-token context was configured; the prompts above do not test a full 16K context.

What do these results tell you?

This is a baseline for local text generation with one active request. It helps estimate response speed at the tested prompt lengths. Prompt caching and thinking were disabled; the model stayed loaded between runs.

It does not establish answer quality, coding accuracy, vision performance, cold-start time or multi-user throughput. Different quantization, context lengths and runtime optimizations need separate measurements.

Estimate memory on another GPU or Mac → Estimates are separate from the measured allocation above.

Tested settings and reproducibility

Hardware: RTX 5090 (32 GB). Runtime: llama.cpp commit 9725a31 with CUDA.

Artifact: gemma-4-12b-it-qat-q4_0.gguf, pinned revision 29d097773436b69ff9feafd636ab4cf873786537. SHA-256: 93567e57a8fe10b23569b9d9ec38cd005deedf71e29477c421a4b83f418a538b.

Server configuration (use a free local port and the path to your verified model file):

llama-server --model gemma-4-12b-it-qat-q4_0.gguf --alias gemma-4-12b-it-qat-q4_0.gguf \
  --host 127.0.0.1 --port 8082 --n-gpu-layers 999 \
  --ctx-size 16384 --parallel 1 --no-cache-prompt --cache-ram 0 \
  --reasoning off --temp 0 --seed 42

Measurements used suite-v1 chat-short and chat-long, with output caps of 256 and 1,024 tokens. The table reports actual token counts. Decode speed and first-token latency use the benchmark client's stream timing.

The client was an isolated copy of llm-speed 0.0.5 with fixes for underscore-containing quantization names and streamed token usage. Prompts and timing formulas were unchanged. Installing the published CLI alone may not reproduce that exact client. The configuration above documents the server; it is not a complete benchmark command.

Five earlier submissions affected by a competing service restart were excluded. Only the three accepted repetitions linked below contribute to this summary.

Inspect the measured runs

Also tested on this GPU: Qwen3.8-27B. Tokenizers and output lengths differ, so speed alone does not rank the models' usefulness.

Measured by llm-speed (@llmspeed) on an RTX 5090 · 22 October 2026 · three repetitions

Gemma 4 on RTX 5090: what changes at 95K prompt tokens?

At 95,212 actual prompt tokens, median first-token delay was 21.54 seconds and generation ran at 117.56 tok/s with Q4_0 quantization.

Compared with 23,872 prompt tokens, the first-token wait grew 5.3× while decode speed fell 10%. This configuration handled all nine measured points with prompt caching disabled.

Gemma 4 Q4_0 on an RTX 5090, measured by llm-speed on 22 October 2026: at 23,872, 47,652 and 95,212 input tokens, median first-token waits were 4.05, 8.70 and 21.54 seconds; decode speeds were 130.15, 127.12 and 117.56 tokens per second. Bars show observed minimum–maximum from three repetitions.

October 22 rerun, measured by llm-speed (@llmspeed) on an RTX 5090: a longer first-token wait can accompany fast generation. Separate axes start at zero; three repetitions describe this configuration, not a confidence interval or a quality score.

Download chart (PNG)Download chart (SVG)Download citationAll nine measurements (JSON)

Reuse under CC BY 4.0. Credit llm-speed and link to the study, test date and configuration. Identify any changes you make to the chart.

Median of three sequential runs; parentheses show observed minimum–maximum. Each row uses actual token usage, not estimated workload targets.

Scroll the table sideways for all measurements.

Long-context medians and observed ranges from three repetitions, 22 October 2026 rerun.
Actual prompt tokensFirst token (s)Decode tok/sOutput tokens
23,8724.052 (4.001–4.059)130.15 (129.82–130.56)98
47,6528.698 (8.674–8.731)127.12 (127.03–127.19)100
95,21221.540 (21.502–21.566)117.56 (117.18–117.59)96

The server had 131,072 tokens of context configured. Workload targets of 32K / 64K / 128K use a character-count estimate; actual prompts were shorter. This is not a full 128K-input test or an answer-quality evaluation. Output cap was 256 tokens, with actual outputs shown above. Three repeats describe this setup, not a confidence interval.

The original September 6 runs (community-submitted, not measured by llm-speed)

This rerun repeats three community-submitted runs from 6 September 2026 with the same llama.cpp commit, model file, context and prompts. Decode speed was within about 5% of the originals (−4.5% / −4.4% / +1.4% at 23,872 / 47,652 / 95,212 tokens). Time to first token was 8–21% longer in the rerun, most likely because it ran on a different machine.

September 6 community-submitted long-context medians and observed ranges.
Actual prompt tokensFirst token (s)Decode tok/sOutput tokens
23,8723.355 (3.334–3.964)136.27 (134.18–142.61)98
47,6527.901 (7.123–8.833)133.03 (131.71–137.28)100
95,21219.901 (19.765–20.224)115.98 (115.05–123.65)96

Original machine: RTX 5090 (32 GB), model kept warm in one session, other idle GPU services resident, driver not recorded. Client 0.0.5 with fixes for streamed token usage and quantization parsing, from an internal revision not available in the public CLI repository. These runs are community-submitted and do not feed llm-speed verdicts.

Long-context settings and reproduction

Rerun machine: RTX 5090 (32 GB), Linux, NVIDIA driver 595.91.07; the card's power and clock limits were not recorded. CUDA runtime: llama.cpp 9725a313b (build b8931, the same commit as the September 6 runs). One active request, all 49 layers on GPU, thinking off; the server was restarted for each repetition.

Pinned artifact: gemma-4-12b-it-qat-q4_0.gguf. SHA-256: 93567e57a8fe10b23569b9d9ec38cd005deedf71e29477c421a4b83f418a538b, checked on the machine before the runs.

Peak device memory was 10,288 MiB during the rerun. The September 6 runs reported a 9,685 MiB model, context and compute allocation, which is not a peak whole-device figure. The 16,384-context chat test above used a different allocation.

Build the pinned runtime with CUDA, verify the artifact, then start this server on a free port. Replace the model filename with its local path:

llama-server --model gemma-4-12b-it-qat-q4_0.gguf --alias gemma-4-12b-it-qat-q4_0.gguf \
  --host 127.0.0.1 --port 8082 --n-gpu-layers 999 \
  --ctx-size 131072 --parallel 1 --no-cache-prompt --cache-ram 0 \
  --reasoning off --temp 0 --seed 42

The rerun used the public llm-speed 0.0.9 client with --strict-anon and unchanged suite-v1 long-context-decay prompts and timing, three sequential invocations:

LLAMA_CPP_SERVER_URL=http://127.0.0.1:8082 \
  llm-speed bench --backend llama.cpp \
  --model /path/to/gemma-4-12b-it-qat-q4_0.gguf --workload long-context-decay

Keep competing workloads idle and preserve normal services. Different drivers, clocks, system load and runtime builds can change timings. The configured capacity does not establish the model's maximum usable context.

Sources and reusable data

Download the dated October 22 rerun: all nine measurements (CSV) or data, configuration and citation (JSON). The September 6 community snapshot stays available: CSV or JSON. Cite this long-context study with its test date and configuration.

Signed runs (16 workload results)

7 of 10 runs were measured by llm-speed (@llmspeed); the other 3 are community-submitted (signed on the submitter's own machine, not measured by llm-speed, and not used for llm-speed verdicts).

12 results · fastest chat-short decode 141.1 tok/s at Q4_0
WorkloadBackendQuantdecode tok/sprefill tok/sTTFTRunSource
chat-shortllama.cpp commit 9725a31Q4_0137.3tok/s2,316.4tok/s52.2msr_8xmm65n1fhqMeasured by llm-speed@llmspeed
chat-longllama.cpp commit 9725a31Q4_0135.6tok/s5,523.3tok/s577msr_8xmm65n1fhqMeasured by llm-speed@llmspeed
chat-shortllama.cpp commit 9725a31Q4_0139.5tok/s1,958.9tok/s61.8msr_c2podhe9qwcMeasured by llm-speed@llmspeed
chat-longllama.cpp commit 9725a31Q4_0137.1tok/s5,118.0tok/s623msr_c2podhe9qwcMeasured by llm-speed@llmspeed
chat-shortllama.cpp commit 9725a31Q4_0141.1tok/s2,027.9tok/s59.7msr_v5arqmazf31Measured by llm-speed@llmspeed
chat-longllama.cpp commit 9725a31Q4_0135.6tok/s5,147.7tok/s619msr_v5arqmazf31Measured by llm-speed@llmspeed
long-context-decayllama.cpp commit 9725a31Q4_0 · 23,872-token prompt134.2tok/s6,022.4tok/s3,964msr_8q-uf-rq-gpCommunity-submitted
long-context-decayllama.cpp commit 9725a31Q4_0 · 23,872-token prompt136.3tok/s7,161.0tok/s3,334msr_1srqmpjnkgdCommunity-submitted
long-context-decayllama.cpp commit 9725a31Q4_0 · 23,872-token prompt142.6tok/s7,115.8tok/s3,355msr_-lh-vczf7ikCommunity-submitted
long-context-decayllama.cpp@8931 (9725a313b)Q4_0 · 23,872-token prompt130.6tok/s5,881.5tok/s4,059msr_9suyze4n7lxMeasured by llm-speed@llmspeed
long-context-decayllama.cpp@8931 (9725a313b)Q4_0 · 23,872-token prompt130.2tok/s5,891.1tok/s4,052msr_fzn_lluvw14Measured by llm-speed@llmspeed
long-context-decayllama.cpp@8931 (9725a313b)Q4_0 · 23,872-token prompt129.8tok/s5,965.8tok/s4,001msr_kdmexe0lkqnMeasured by llm-speed@llmspeed

RTX 4090 (24GB)RTX 4090 (24GB)measured by llm-speed

4 results · fastest chat-short decode 97.7 tok/s at Q4_0
WorkloadBackendQuantdecode tok/sprefill tok/sTTFTRunSource
chat-shortllama.cpp@8931 (9725a313b)Q4_097.72tok/s1,109.0tok/s112msr_jfh0qkngw76Measured by llm-speed@llmspeed
chat-longllama.cpp@8931 (9725a313b)Q4_086.30tok/s1,743.0tok/s1,831msr_jfh0qkngw76Measured by llm-speed@llmspeed
concurrent-decodellama.cpp@8931 (9725a313b)Q4_095.49tok/sno datano datar_jfh0qkngw76Measured by llm-speed@llmspeed
agent-tracellama.cpp@8931 (9725a313b)Q4_089.10tok/s2,305.0tok/s983msr_jfh0qkngw76Measured by llm-speed@llmspeed

Gemma 4 12B IT QAT on hardware

Request a gemma-4-12b-it-qat run

No measured run yet for these pairs. Vote for the one you want; the most-requested runs go first. See all requests.

0 votes

We never sell or share your email. Privacy

llm-speed can measure this card · Submit your own

Check it for your setup

  • Predict your exact setup: estimate whether Gemma 4 12B IT QAT fits your GPU or Mac at your quant and context, next to measured speeds where we have them.
  • Request this pair: ask for a measured Gemma 4 12B IT QAT run on hardware we have not measured. Requests are ranked by votes.
  • Full run table (10 runs): every Gemma 4 12B IT QAT result we have, with links to each source run.