RTX 5090 (32GB) LLM benchmark

The fastest LLM measured on the RTX 5090 (32GB) is stable-code-instruct-3b at 356.1 decode tok/s via llama.cpp (signed run). Across 103 reproducible runs on 21 models, this page lists decode tok/s, prefill, and TTFT for each, every number linking to the run it came from.

Fastest known config on RTX 5090 (32GB)

356.1 decode tok/s

stable-code-instruct-3b via llama.cpp. see full run

Original measurements · 5 September 2026

Gemma 4 12B vs Qwen3.8-27B on RTX 5090

Compare response speed, first-token delay and GPU allocation on the same rig. Both tested model files fit entirely on this 32 GB GPU with a 16,384-token context configured. The prompts below are shorter than that configured limit.

Gemma 4 12B IT QAT

Q4_0 · 6.98 GB model file

7.75 GiB runtime GPU allocation

Model + context + compute buffers; all 49 layers on GPU.

Settings and three source runs →

Qwen3.8-27B

Q4_K_M · 18.97 GB model file

18.62 GiB runtime GPU allocation

Model + context + compute buffers; all 65 layers on GPU.

Settings and three source runs →

Medians of three repetitions per model and workload. Parentheses show observed minimum–maximum, not confidence intervals. Scroll the table on small screens.
Model / workload	Decode tok/s ↑	First token ms ↓	Actual input tokens	Actual output tokens
Gemma 4 12B IT QATQ4_0 · chat-short	139.5(137.3–141.1)	59.7(52.2–61.8)	121	256
Qwen3.8-27BQ4_K_M · chat-short	65.7(65.4–65.7)	117.5(117.1–131.7)	114	256
Gemma 4 12B IT QATQ4_0 · chat-long	135.6(135.6–137.1)	619.3(577.2–622.9)	3,188	620
Qwen3.8-27BQ4_K_M · chat-long	65.1(65.1–65.1)	909.4(908.0–910.5)	3,184	874

Same llama.cpp CUDA runtime, suite-v1 text prompts, one request at a time, warm models, thinking and prompt caching off. Output caps: 256 tokens for chat-short and 1,024 for chat-long. Tokenizers and actual output lengths differ.

Use these results to shortlist models for your latency and memory needs, then test answer quality on your own tasks. Coding accuracy, reasoning quality, vision and multi-user capacity were not measured. Runtime allocation is not peak whole-device memory; other idle services remained GPU-resident.

RTX 5090 model studies

Repeated tests with pinned model files, runtime settings, observed ranges and source runs. Choose a study for configuration details beyond the leaderboard's single result.

gemma-4-12b-it-qat

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
long-context-decay	llama.cpp@1 (9725a31)	Q4_0	142.6tok/s	7,115.8tok/s	3,355ms	r_-lh-vczf7ik
long-context-decay	llama.cpp@1 (9725a31)	Q4_0	136.3tok/s	7,161.0tok/s	3,334ms	r_1srqmpjnkgd
long-context-decay	llama.cpp@1 (9725a31)	Q4_0	134.2tok/s	6,022.4tok/s	3,964ms	r_8q-uf-rq-gp
chat-short	llama.cpp@1 (9725a31)	Q4_0	141.1tok/s	2,027.9tok/s	59.7ms	r_v5arqmazf31
chat-long	llama.cpp@1 (9725a31)	Q4_0	135.6tok/s	5,147.7tok/s	619ms	r_v5arqmazf31
chat-short	llama.cpp@1 (9725a31)	Q4_0	139.5tok/s	1,958.9tok/s	61.8ms	r_c2podhe9qwc
chat-long	llama.cpp@1 (9725a31)	Q4_0	137.1tok/s	5,118.0tok/s	623ms	r_c2podhe9qwc
chat-short	llama.cpp@1 (9725a31)	Q4_0	137.3tok/s	2,316.4tok/s	52.2ms	r_8xmm65n1fhq
chat-long	llama.cpp@1 (9725a31)	Q4_0	135.6tok/s	5,523.3tok/s	577ms	r_8xmm65n1fhq

Qwen3.8-27B

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp@1 (9725a31)	Q4_K_M	65.66tok/s	969.8tok/s	118ms	r_dz5a3lzxhrj
chat-long	llama.cpp@1 (9725a31)	Q4_K_M	65.10tok/s	3,496.8tok/s	911ms	r_dz5a3lzxhrj
chat-short	llama.cpp@1 (9725a31)	Q4_K_M	65.73tok/s	973.6tok/s	117ms	r_c3ps9wjygi9
chat-long	llama.cpp@1 (9725a31)	Q4_K_M	65.09tok/s	3,506.5tok/s	908ms	r_c3ps9wjygi9
chat-short	llama.cpp@1 (9725a31)	Q4_K_M	65.43tok/s	865.3tok/s	132ms	r_7mod38qsldj
chat-long	llama.cpp@1 (9725a31)	Q4_K_M	65.10tok/s	3,501.1tok/s	909ms	r_7mod38qsldj

Coder-V2-Lite-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	293.1tok/s	no data	297ms	r_bfpto9so2o1
chat-long	llama.cpp	-	181.2tok/s	no data	361ms	r_bfpto9so2o1
concurrent-decode	llama.cpp	-	268.4tok/s	no data	no data	r_bfpto9so2o1
agent-trace	llama.cpp	-	204.5tok/s	24,978.7tok/s	80.7ms	r_bfpto9so2o1
chat-short	llama.cpp	-	309.5tok/s	no data	268ms	r_0_gs1rgl2fl

gpt-oss-20b

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	318.4tok/s	no data	173ms	r_b9ul-vxh9sc
chat-long	llama.cpp	-	301.6tok/s	no data	239ms	r_b9ul-vxh9sc
concurrent-decode	llama.cpp	-	302.8tok/s	no data	no data	r_b9ul-vxh9sc
agent-trace	llama.cpp	-	304.5tok/s	26,907.5tok/s	77.7ms	r_b9ul-vxh9sc
chat-short	llama.cpp	-	69.42tok/s	no data	333ms	r_r9h57uts9lr

Yi-Coder-9B-Chat

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	66.79tok/s	no data	317ms	r_zlh6az5q0o_
chat-long	llama.cpp	-	69.60tok/s	no data	1,195ms	r_zlh6az5q0o_
concurrent-decode	llama.cpp	-	70.99tok/s	no data	no data	r_zlh6az5q0o_
agent-trace	llama.cpp	-	71.76tok/s	4,478.0tok/s	397ms	r_zlh6az5q0o_
chat-short	llama.cpp	-	199.3tok/s	no data	36.2ms	r_u4iojm6-ekg

stable-code-instruct-3b

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	341.9tok/s	no data	175ms	r_q9f15lz6831
chat-long	llama.cpp	-	289.1tok/s	no data	232ms	r_q9f15lz6831
concurrent-decode	llama.cpp	-	356.1tok/s	no data	no data	r_q9f15lz6831
agent-trace	llama.cpp	-	292.8tok/s	44,387.5tok/s	42.7ms	r_q9f15lz6831
chat-short	llama.cpp	-	331.0tok/s	no data	36.7ms	r_8l57cim1i10

Qwen2.5-32B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	68.71tok/s	no data	77.0ms	r_bjy5a5izxjc
chat-long	llama.cpp	-	65.36tok/s	no data	968ms	r_bjy5a5izxjc
concurrent-decode	llama.cpp	-	68.60tok/s	no data	no data	r_bjy5a5izxjc
agent-trace	llama.cpp	-	66.45tok/s	10,407.9tok/s	214ms	r_bjy5a5izxjc
chat-short	llama.cpp	-	71.90tok/s	no data	115ms	r_twfs86tf_xf

Qwen2.5-14B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	130.3tok/s	no data	48.6ms	r_xr4qdv1hgf2
chat-long	llama.cpp	-	122.7tok/s	no data	489ms	r_xr4qdv1hgf2
concurrent-decode	llama.cpp	-	131.4tok/s	no data	no data	r_xr4qdv1hgf2
agent-trace	llama.cpp	-	122.8tok/s	20,315.4tok/s	111ms	r_xr4qdv1hgf2
chat-short	llama.cpp	-	133.3tok/s	no data	55.9ms	r_tj9bu7gvnvh

Llama-3.1-8B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	229.4tok/s	no data	35.6ms	r_qr4srge34da
chat-long	llama.cpp	-	211.4tok/s	no data	266ms	r_qr4srge34da
concurrent-decode	llama.cpp	-	223.0tok/s	no data	no data	r_qr4srge34da
agent-trace	llama.cpp	-	210.3tok/s	36,720.3tok/s	60.8ms	r_qr4srge34da
chat-short	llama.cpp	-	232.2tok/s	no data	33.8ms	r_kfrkg-vn376

Qwen2.5-7B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	232.7tok/s	no data	37.0ms	r_1shiviswt3d
chat-long	llama.cpp	-	232.4tok/s	no data	239ms	r_1shiviswt3d
concurrent-decode	llama.cpp	-	240.7tok/s	no data	no data	r_1shiviswt3d
agent-trace	llama.cpp	-	232.5tok/s	38,007.7tok/s	57.1ms	r_1shiviswt3d
chat-short	llama.cpp	-	246.7tok/s	no data	33.9ms	r_3yn-4321hp-

Qwen3-32B

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	66.51tok/s	no data	158ms	r_-txe_hiq44n
chat-long	llama.cpp	-	62.30tok/s	no data	1,067ms	r_-txe_hiq44n
concurrent-decode	llama.cpp	-	66.64tok/s	no data	no data	r_-txe_hiq44n
agent-trace	llama.cpp	-	64.34tok/s	8,394.1tok/s	259ms	r_-txe_hiq44n
chat-short	llama.cpp	-	69.45tok/s	no data	327ms	r_phvxm9dcak0

phi-4

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	139.1tok/s	no data	51.1ms	r_k1u_k1j_1i2
chat-long	llama.cpp	-	133.7tok/s	no data	433ms	r_k1u_k1j_1i2
concurrent-decode	llama.cpp	-	140.2tok/s	no data	no data	r_k1u_k1j_1i2
agent-trace	llama.cpp	-	124.8tok/s	21,162.4tok/s	102ms	r_k1u_k1j_1i2
chat-short	llama.cpp	-	140.9tok/s	no data	56.7ms	r_e-k4aea8ipr

gemma-2-9b-it

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	69.45tok/s	no data	325ms	r_1_xl4zb5-xj
chat-long	llama.cpp	-	69.42tok/s	no data	1,044ms	r_1_xl4zb5-xj
concurrent-decode	llama.cpp	-	68.57tok/s	no data	no data	r_1_xl4zb5-xj
agent-trace	llama.cpp	-	68.46tok/s	4,816.8tok/s	420ms	r_1_xl4zb5-xj
chat-short	llama.cpp	-	152.7tok/s	no data	58.9ms	r__b_bzmmab_8

Qwen2.5-Coder-32B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	67.41tok/s	no data	384ms	r_v983y0y3r2u
chat-long	llama.cpp	-	69.15tok/s	no data	1,251ms	r_v983y0y3r2u
concurrent-decode	llama.cpp	-	70.96tok/s	no data	no data	r_v983y0y3r2u
agent-trace	llama.cpp	-	68.56tok/s	4,677.1tok/s	414ms	r_v983y0y3r2u
chat-short	llama.cpp	-	71.91tok/s	no data	71.0ms	r_nkbs6d3-d21

Qwen2.5-Coder-14B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	140.0tok/s	no data	42.1ms	r_opuj21f13-_
chat-long	llama.cpp	-	123.0tok/s	no data	457ms	r_opuj21f13-_
concurrent-decode	llama.cpp	-	133.8tok/s	no data	no data	r_opuj21f13-_
agent-trace	llama.cpp	-	130.5tok/s	21,985.9tok/s	104ms	r_opuj21f13-_
chat-short	llama.cpp	-	136.0tok/s	no data	51.3ms	r_p_f63tcgans

Qwen2.5-Coder-7B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	255.3tok/s	no data	35.4ms	r_2b9o6y_49mi
chat-long	llama.cpp	-	244.0tok/s	no data	228ms	r_2b9o6y_49mi
concurrent-decode	llama.cpp	-	254.0tok/s	no data	no data	r_2b9o6y_49mi
agent-trace	llama.cpp	-	229.9tok/s	38,735.2tok/s	57.4ms	r_2b9o6y_49mi
chat-short	llama.cpp	-	244.9tok/s	no data	36.1ms	r_mln72x5zbis

Qwen3-Coder-30B-A3B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	68.41tok/s	no data	334ms	r_pm_a1uf2ufc
chat-long	llama.cpp	-	70.55tok/s	no data	1,204ms	r_pm_a1uf2ufc
concurrent-decode	llama.cpp	-	68.04tok/s	no data	no data	r_pm_a1uf2ufc
agent-trace	llama.cpp	-	66.09tok/s	4,816.7tok/s	451ms	r_pm_a1uf2ufc
chat-short	llama.cpp	-	259.9tok/s	no data	218ms	r_c7qyvvmmsv1

Codestral-22B-v0.1

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	93.72tok/s	no data	68.6ms	r_4q040m4scic
chat-long	llama.cpp	-	87.36tok/s	no data	849ms	r_4q040m4scic
concurrent-decode	llama.cpp	-	100.3tok/s	no data	no data	r_4q040m4scic
agent-trace	llama.cpp	-	93.44tok/s	10,351.0tok/s	191ms	r_4q040m4scic
chat-short	llama.cpp	-	69.95tok/s	no data	199ms	r_sqr8liqh4ii

Qwen3.6-35B-A3B-Q4_K_M.gguf

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	223.3tok/s	no data	263ms	r_dp6sr2_iwcf
chat-short	llama.cpp	-	224.0tok/s	no data	167ms	r_a56-wxl21lk
chat-short	llama.cpp	-	216.5tok/s	no data	185ms	r_k070lz99uzi

gemma-4-31B-it-Q4_K_M.gguf

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	67.37tok/s	no data	180ms	r_uut6m_v6ui9
chat-short	llama.cpp	-	67.27tok/s	no data	344ms	r_23fireoga9y

Qwen3.6-27B-Q4_K_M.gguf

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	72.38tok/s	no data	167ms	r_yluotk909p8
chat-short	llama.cpp	-	72.48tok/s	no data	172ms	r_v2vrkr4uah1
chat-short	llama.cpp	-	69.56tok/s	no data	227ms	r_u4wa_y_y3vt

Community folklore on RTX 5090 (32GB)

134 unverified claims extracted from Reddit/HN comments. Lower trust than signed runs above; every row links to the source.

communityconfidence 75%
10.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“95fa52dbf8ebec6acaf0105e1e9 Hey all, Just a quick one in case it saves someone else a headache. I was getting really poor throughput (\~10 tok/sec) with Qwen3-Coder-Next-Q4\_K\_S.gguf on llama.cpp, like “this can’t be right” levels, and eventually found a set of args that fix…”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
26.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“~26 tok/sec with Unsloth Qwen3-Coder-Next-Q4_K_S on RTX 5090 (Windows/llama.cpp) https://preview.redd.it/9gfytpz5srhg1.png?width=692&format=png&auto=w”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
10.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“95fa52dbf8ebec6acaf0105e1e9 Hey all, Just a quick one in case it saves someone else a headache. I was getting really poor throughput (\~10 tok/sec) with Qwen3-Coder-Next-Q4\_K\_S.gguf on llama.cpp, like “this can’t be right” levels, and eventually found a set of args that fix…”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
26.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“~26 tok/sec with Unsloth Qwen3-Coder-Next-Q4_K_S on RTX 5090 (Windows/llama.cpp) https://preview.redd.it/9gfytpz5srhg1.png?width=692&format=png&auto=w”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
10.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“95fa52dbf8ebec6acaf0105e1e9 Hey all, Just a quick one in case it saves someone else a headache. I was getting really poor throughput (\~10 tok/sec) with Qwen3-Coder-Next-Q4\_K\_S.gguf on llama.cpp, like “this can’t be right” levels, and eventually found a set of args that fix…”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
26.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“~26 tok/sec with Unsloth Qwen3-Coder-Next-Q4_K_S on RTX 5090 (Windows/llama.cpp) https://preview.redd.it/9gfytpz5srhg1.png?width=692&format=png&auto=w”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
207.9tok/s — Qwen3-Coder on RTX 5090 via sglang AWQ
our signed data: RTX 5090 · Qwen3-Coder
“hoosing the Framework **RTX 5090 — Qwen3-Coder-30B-A3B-Instruct-AWQ** |Metric|vLLM|SGLang| |:-|:-|:-| |Output throughput|**555.82 tok/s**|207.93 tok/s| |Mean TTFT|**549 ms**|1,558 ms| |Median TPOT|**7.06 ms**|18.84 ms| vLLM wins by 2.7x. SGLang is required `--quantization moe_…”
source: Reddit · u/NoVibeCoding · 2026-03-06
communityconfidence 75%
555.8tok/s — Qwen3-Coder on RTX 5090 via sglang AWQ
our signed data: RTX 5090 · Qwen3-Coder
“atency? # 1. Choosing the Framework **RTX 5090 — Qwen3-Coder-30B-A3B-Instruct-AWQ** |Metric|vLLM|SGLang| |:-|:-|:-| |Output throughput|**555.82 tok/s**|207.93 tok/s| |Mean TTFT|**549 ms**|1,558 ms| |Median TPOT|**7.06 ms**|18.84 ms| vLLM wins by 2.7x. SGLang is required `--qu…”
source: Reddit · u/NoVibeCoding · 2026-03-06
communityconfidence 75%
207.9tok/s — Qwen3-Coder on RTX 5090 via sglang AWQ
our signed data: RTX 5090 · Qwen3-Coder
“hoosing the Framework **RTX 5090 — Qwen3-Coder-30B-A3B-Instruct-AWQ** |Metric|vLLM|SGLang| |:-|:-|:-| |Output throughput|**555.82 tok/s**|207.93 tok/s| |Mean TTFT|**549 ms**|1,558 ms| |Median TPOT|**7.06 ms**|18.84 ms| vLLM wins by 2.7x. SGLang is required `--quantization moe_…”
source: Reddit · u/NoVibeCoding · 2026-03-06
communityconfidence 75%
555.8tok/s — Qwen3-Coder on RTX 5090 via sglang AWQ
our signed data: RTX 5090 · Qwen3-Coder
“atency? # 1. Choosing the Framework **RTX 5090 — Qwen3-Coder-30B-A3B-Instruct-AWQ** |Metric|vLLM|SGLang| |:-|:-|:-| |Output throughput|**555.82 tok/s**|207.93 tok/s| |Mean TTFT|**549 ms**|1,558 ms| |Median TPOT|**7.06 ms**|18.84 ms| vLLM wins by 2.7x. SGLang is required `--qu…”
source: Reddit · u/NoVibeCoding · 2026-03-06

See all 134 claims for RTX 5090 (32GB) →

Common questions about RTX 5090 (32GB)

Direct Q&A drawn from the runs above: fastest LLM, supported model classes, backend rankings, quantization guidance.

Read the RTX 5090 (32GB) FAQ →

Gemma 4 12B vs Qwen3.8-27B on RTX 5090

Gemma 4 12B IT QAT

Qwen3.8-27B

RTX 5090 model studies

Community folklore on RTX 5090 (32GB)

Models measured on RTX 5090 (32GB)

Common questions about RTX 5090 (32GB)