Gemma 4 12B IT QAT
6 workload results across 1 hardware configuration.
Original test · 5 September 2026 · three repetitions
How fast is Gemma 4 12B IT QAT on an RTX 5090?
We measured 139.5 tokens per second in short chat with Q4_0 quantization, using one RTX 5090 with 32 GB VRAM and llama.cpp. Median time to the first token was 59.7 ms with the model already loaded. These are single-user, thinking-off results.
| Workload | Decode tok/s | First token (ms) | Prompt tokens | Output tokens |
|---|---|---|---|---|
| chat-short | 139.5 (137.3–141.1) | 59.7 (52.2–61.8) | 121 | 256 |
| chat-long | 135.6 (135.6–137.1) | 619.3 (577.2–622.9) | 3,188 | 620 |
Does it fit in 32 GB of VRAM?
Yes, this tested configuration offloaded all 49/49 layers to the GPU. The runtime reported 7.75 GiB (7,931 MiB) of GPU allocation for model, context and compute buffers combined. The model file is 6.98 GB on disk.
Allocation is from the runtime log, not a measurement of peak whole-device memory. A 16,384-token context was configured; the prompts above do not test a full 16K context. Other idle services remained GPU-resident.
What do these results tell you?
This is a baseline for local text generation with one active request. It helps estimate response speed at the tested prompt lengths. Prompt caching and thinking were disabled; the model stayed loaded between runs.
It does not establish answer quality, coding accuracy, vision performance, cold-start time or multi-user throughput. Different quantization, context lengths and runtime optimizations need separate measurements.
Tested settings and reproducibility
Hardware: RTX 5090 (32 GB), AMD Ryzen 7 9850X3D, 30 GB system RAM reported by WSL. Runtime: llama.cpp commit 9725a31 with CUDA.
Artifact: gemma-4-12b-it-qat-q4_0.gguf, pinned revision 29d097773436b69ff9feafd636ab4cf873786537. SHA-256: 93567e57a8fe10b23569b9d9ec38cd005deedf71e29477c421a4b83f418a538b.
Server configuration (use a free local port and the path to your verified model file):
llama-server --model gemma-4-12b-it-qat-q4_0.gguf --alias gemma-4-12b-it-qat-q4_0.gguf \
--host 127.0.0.1 --port 8082 --n-gpu-layers 999 \
--ctx-size 16384 --parallel 1 --no-cache-prompt --cache-ram 0 \
--reasoning off --temp 0 --seed 42Measurements used suite-v1 chat-short and chat-long, with output caps of 256 and 1,024 tokens. The table reports actual token counts. Decode speed and first-token latency use the benchmark client's stream timing.
The client was an isolated copy of llm-speed 0.0.5 with fixes for underscore-containing quantization names and streamed token usage. Prompts and timing formulas were unchanged. Installing the published CLI alone may not reproduce that exact client. The configuration above documents the server; it is not a complete benchmark command.
Five earlier submissions affected by a competing service restart were excluded. Only the three accepted repetitions linked below contribute to this summary.
Inspect the measured runs
Also tested on this GPU: Qwen3.8-27B. Tokenizers and output lengths differ, so speed alone does not rank the models' usefulness.
Local runs (6 runs)
Runs from contributors' own machines via MLX, llama.cpp, vLLM, exllamav2, or ollama. Signed on the submitter's hardware.
RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp@1 (9725a31) | Q4_0 | 141.1tok/s | 2,027.9tok/s | 59.7ms | r_v5arqmazf31 |
| chat-long | llama.cpp@1 (9725a31) | Q4_0 | 135.6tok/s | 5,147.7tok/s | 619ms | r_v5arqmazf31 |
| chat-short | llama.cpp@1 (9725a31) | Q4_0 | 139.5tok/s | 1,958.9tok/s | 61.8ms | r_c2podhe9qwc |
| chat-long | llama.cpp@1 (9725a31) | Q4_0 | 137.1tok/s | 5,118.0tok/s | 623ms | r_c2podhe9qwc |
| chat-short | llama.cpp@1 (9725a31) | Q4_0 | 137.3tok/s | 2,316.4tok/s | 52.2ms | r_8xmm65n1fhq |
| chat-long | llama.cpp@1 (9725a31) | Q4_0 | 135.6tok/s | 5,523.3tok/s | 577ms | r_8xmm65n1fhq |