RTX 5090 (32GB) LLM benchmark

The fastest LLM measured on the RTX 5090 (32GB) is stable-code-instruct-3b at 356.1 decode tok/s via llama.cpp (signed run). Across 97 reproducible runs on 19 models, this page lists decode tok/s, prefill, and TTFT for each, every number linking to the run it came from.

Fastest known config on RTX 5090 (32GB)

356.1 decode tok/s

stable-code-instruct-3b via llama.cpp. see full run

Coder-V2-Lite-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	293.1tok/s	no data	297ms	r_bfpto9so2o1
chat-long	llama.cpp	-	181.2tok/s	no data	361ms	r_bfpto9so2o1
concurrent-decode	llama.cpp	-	268.4tok/s	no data	no data	r_bfpto9so2o1
agent-trace	llama.cpp	-	204.5tok/s	24,978.7tok/s	80.7ms	r_bfpto9so2o1
chat-short	llama.cpp	-	309.5tok/s	no data	268ms	r_0_gs1rgl2fl

gpt-oss-20b

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	318.4tok/s	no data	173ms	r_b9ul-vxh9sc
chat-long	llama.cpp	-	301.6tok/s	no data	239ms	r_b9ul-vxh9sc
concurrent-decode	llama.cpp	-	302.8tok/s	no data	no data	r_b9ul-vxh9sc
agent-trace	llama.cpp	-	304.5tok/s	26,907.5tok/s	77.7ms	r_b9ul-vxh9sc
chat-short	llama.cpp	-	69.42tok/s	no data	333ms	r_r9h57uts9lr

Yi-Coder-9B-Chat

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	66.79tok/s	no data	317ms	r_zlh6az5q0o_
chat-long	llama.cpp	-	69.60tok/s	no data	1,195ms	r_zlh6az5q0o_
concurrent-decode	llama.cpp	-	70.99tok/s	no data	no data	r_zlh6az5q0o_
agent-trace	llama.cpp	-	71.76tok/s	4,478.0tok/s	397ms	r_zlh6az5q0o_
chat-short	llama.cpp	-	199.3tok/s	no data	36.2ms	r_u4iojm6-ekg

stable-code-instruct-3b

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	341.9tok/s	no data	175ms	r_q9f15lz6831
chat-long	llama.cpp	-	289.1tok/s	no data	232ms	r_q9f15lz6831
concurrent-decode	llama.cpp	-	356.1tok/s	no data	no data	r_q9f15lz6831
agent-trace	llama.cpp	-	292.8tok/s	44,387.5tok/s	42.7ms	r_q9f15lz6831
chat-short	llama.cpp	-	331.0tok/s	no data	36.7ms	r_8l57cim1i10

Qwen2.5-32B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	68.71tok/s	no data	77.0ms	r_bjy5a5izxjc
chat-long	llama.cpp	-	65.36tok/s	no data	968ms	r_bjy5a5izxjc
concurrent-decode	llama.cpp	-	68.60tok/s	no data	no data	r_bjy5a5izxjc
agent-trace	llama.cpp	-	66.45tok/s	10,407.9tok/s	214ms	r_bjy5a5izxjc
chat-short	llama.cpp	-	71.90tok/s	no data	115ms	r_twfs86tf_xf

Qwen2.5-14B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	130.3tok/s	no data	48.6ms	r_xr4qdv1hgf2
chat-long	llama.cpp	-	122.7tok/s	no data	489ms	r_xr4qdv1hgf2
concurrent-decode	llama.cpp	-	131.4tok/s	no data	no data	r_xr4qdv1hgf2
agent-trace	llama.cpp	-	122.8tok/s	20,315.4tok/s	111ms	r_xr4qdv1hgf2
chat-short	llama.cpp	-	133.3tok/s	no data	55.9ms	r_tj9bu7gvnvh

Llama-3.1-8B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	229.4tok/s	no data	35.6ms	r_qr4srge34da
chat-long	llama.cpp	-	211.4tok/s	no data	266ms	r_qr4srge34da
concurrent-decode	llama.cpp	-	223.0tok/s	no data	no data	r_qr4srge34da
agent-trace	llama.cpp	-	210.3tok/s	36,720.3tok/s	60.8ms	r_qr4srge34da
chat-short	llama.cpp	-	232.2tok/s	no data	33.8ms	r_kfrkg-vn376

Qwen2.5-7B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	232.7tok/s	no data	37.0ms	r_1shiviswt3d
chat-long	llama.cpp	-	232.4tok/s	no data	239ms	r_1shiviswt3d
concurrent-decode	llama.cpp	-	240.7tok/s	no data	no data	r_1shiviswt3d
agent-trace	llama.cpp	-	232.5tok/s	38,007.7tok/s	57.1ms	r_1shiviswt3d
chat-short	llama.cpp	-	246.7tok/s	no data	33.9ms	r_3yn-4321hp-

Qwen3-32B

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	66.51tok/s	no data	158ms	r_-txe_hiq44n
chat-long	llama.cpp	-	62.30tok/s	no data	1,067ms	r_-txe_hiq44n
concurrent-decode	llama.cpp	-	66.64tok/s	no data	no data	r_-txe_hiq44n
agent-trace	llama.cpp	-	64.34tok/s	8,394.1tok/s	259ms	r_-txe_hiq44n
chat-short	llama.cpp	-	69.45tok/s	no data	327ms	r_phvxm9dcak0

phi-4

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	139.1tok/s	no data	51.1ms	r_k1u_k1j_1i2
chat-long	llama.cpp	-	133.7tok/s	no data	433ms	r_k1u_k1j_1i2
concurrent-decode	llama.cpp	-	140.2tok/s	no data	no data	r_k1u_k1j_1i2
agent-trace	llama.cpp	-	124.8tok/s	21,162.4tok/s	102ms	r_k1u_k1j_1i2
chat-short	llama.cpp	-	140.9tok/s	no data	56.7ms	r_e-k4aea8ipr

gemma-2-9b-it

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	69.45tok/s	no data	325ms	r_1_xl4zb5-xj
chat-long	llama.cpp	-	69.42tok/s	no data	1,044ms	r_1_xl4zb5-xj
concurrent-decode	llama.cpp	-	68.57tok/s	no data	no data	r_1_xl4zb5-xj
agent-trace	llama.cpp	-	68.46tok/s	4,816.8tok/s	420ms	r_1_xl4zb5-xj
chat-short	llama.cpp	-	152.7tok/s	no data	58.9ms	r__b_bzmmab_8

Qwen2.5-Coder-32B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	67.41tok/s	no data	384ms	r_v983y0y3r2u
chat-long	llama.cpp	-	69.15tok/s	no data	1,251ms	r_v983y0y3r2u
concurrent-decode	llama.cpp	-	70.96tok/s	no data	no data	r_v983y0y3r2u
agent-trace	llama.cpp	-	68.56tok/s	4,677.1tok/s	414ms	r_v983y0y3r2u
chat-short	llama.cpp	-	71.91tok/s	no data	71.0ms	r_nkbs6d3-d21

Qwen2.5-Coder-14B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	140.0tok/s	no data	42.1ms	r_opuj21f13-_
chat-long	llama.cpp	-	123.0tok/s	no data	457ms	r_opuj21f13-_
concurrent-decode	llama.cpp	-	133.8tok/s	no data	no data	r_opuj21f13-_
agent-trace	llama.cpp	-	130.5tok/s	21,985.9tok/s	104ms	r_opuj21f13-_
chat-short	llama.cpp	-	136.0tok/s	no data	51.3ms	r_p_f63tcgans

Qwen2.5-Coder-7B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	255.3tok/s	no data	35.4ms	r_2b9o6y_49mi
chat-long	llama.cpp	-	244.0tok/s	no data	228ms	r_2b9o6y_49mi
concurrent-decode	llama.cpp	-	254.0tok/s	no data	no data	r_2b9o6y_49mi
agent-trace	llama.cpp	-	229.9tok/s	38,735.2tok/s	57.4ms	r_2b9o6y_49mi
chat-short	llama.cpp	-	244.9tok/s	no data	36.1ms	r_mln72x5zbis

Qwen3-Coder-30B-A3B-Instruct

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	68.41tok/s	no data	334ms	r_pm_a1uf2ufc
chat-long	llama.cpp	-	70.55tok/s	no data	1,204ms	r_pm_a1uf2ufc
concurrent-decode	llama.cpp	-	68.04tok/s	no data	no data	r_pm_a1uf2ufc
agent-trace	llama.cpp	-	66.09tok/s	4,816.7tok/s	451ms	r_pm_a1uf2ufc
chat-short	llama.cpp	-	259.9tok/s	no data	218ms	r_c7qyvvmmsv1

Codestral-22B-v0.1

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	93.72tok/s	no data	68.6ms	r_4q040m4scic
chat-long	llama.cpp	-	87.36tok/s	no data	849ms	r_4q040m4scic
concurrent-decode	llama.cpp	-	100.3tok/s	no data	no data	r_4q040m4scic
agent-trace	llama.cpp	-	93.44tok/s	10,351.0tok/s	191ms	r_4q040m4scic
chat-short	llama.cpp	-	69.95tok/s	no data	199ms	r_sqr8liqh4ii

Qwen3.6-35B-A3B-Q4_K_M.gguf

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	223.3tok/s	no data	263ms	r_dp6sr2_iwcf
chat-short	llama.cpp	-	224.0tok/s	no data	167ms	r_a56-wxl21lk
chat-short	llama.cpp	-	216.5tok/s	no data	185ms	r_k070lz99uzi
chat-short	llama.cpp	-	221.6tok/s	no data	207ms	r_dqelw_a8f2c

gemma-4-31B-it-Q4_K_M.gguf

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	67.37tok/s	no data	180ms	r_uut6m_v6ui9
chat-short	llama.cpp	-	67.27tok/s	no data	344ms	r_23fireoga9y
chat-short	llama.cpp	-	67.33tok/s	no data	156ms	r_1ofpvf4m7p5
chat-short	llama.cpp	-	67.31tok/s	no data	160ms	r_wwq2l7lmy6d

Qwen3.6-27B-Q4_K_M.gguf

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	72.38tok/s	no data	167ms	r_yluotk909p8
chat-short	llama.cpp	-	72.48tok/s	no data	172ms	r_v2vrkr4uah1
chat-short	llama.cpp	-	69.56tok/s	no data	227ms	r_u4wa_y_y3vt
chat-short	llama.cpp	-	72.57tok/s	no data	172ms	r_l-wg5a6o-vg
chat-short	llama.cpp	-	69.56tok/s	no data	174ms	r_w482rf72v6z
chat-short	llama.cpp	-	72.32tok/s	no data	290ms	r_ercjdbdw2gi
chat-short	llama.cpp	-	69.60tok/s	no data	382ms	r_aj808r0dw53
chat-short	llama.cpp	-	72.31tok/s	no data	198ms	r_4upad7ubcpi
chat-short	llama.cpp	-	68.00tok/s	no data	241ms	r_qiq8q_cqfk5

Community folklore on RTX 5090 (32GB)

134 unverified claims extracted from Reddit/HN comments. Lower trust than signed runs above; every row links to the source.

communityconfidence 75%
10.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“95fa52dbf8ebec6acaf0105e1e9 Hey all, Just a quick one in case it saves someone else a headache. I was getting really poor throughput (\~10 tok/sec) with Qwen3-Coder-Next-Q4\_K\_S.gguf on llama.cpp, like “this can’t be right” levels, and eventually found a set of args that fix…”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
26.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“~26 tok/sec with Unsloth Qwen3-Coder-Next-Q4_K_S on RTX 5090 (Windows/llama.cpp) https://preview.redd.it/9gfytpz5srhg1.png?width=692&format=png&auto=w”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
10.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“95fa52dbf8ebec6acaf0105e1e9 Hey all, Just a quick one in case it saves someone else a headache. I was getting really poor throughput (\~10 tok/sec) with Qwen3-Coder-Next-Q4\_K\_S.gguf on llama.cpp, like “this can’t be right” levels, and eventually found a set of args that fix…”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
26.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“~26 tok/sec with Unsloth Qwen3-Coder-Next-Q4_K_S on RTX 5090 (Windows/llama.cpp) https://preview.redd.it/9gfytpz5srhg1.png?width=692&format=png&auto=w”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
10.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“95fa52dbf8ebec6acaf0105e1e9 Hey all, Just a quick one in case it saves someone else a headache. I was getting really poor throughput (\~10 tok/sec) with Qwen3-Coder-Next-Q4\_K\_S.gguf on llama.cpp, like “this can’t be right” levels, and eventually found a set of args that fix…”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
26.00tok/s — Qwen3-Coder-Next on RTX 5090 via llama.cpp Q4_K_S
our signed data: RTX 5090 · Qwen3-Coder-Next
“~26 tok/sec with Unsloth Qwen3-Coder-Next-Q4_K_S on RTX 5090 (Windows/llama.cpp) https://preview.redd.it/9gfytpz5srhg1.png?width=692&format=png&auto=w”
source: Reddit · u/Spiritual_Tie_5574 · 2026-02-06
communityconfidence 75%
207.9tok/s — Qwen3-Coder on RTX 5090 via sglang AWQ
our signed data: RTX 5090 · Qwen3-Coder
“hoosing the Framework **RTX 5090 — Qwen3-Coder-30B-A3B-Instruct-AWQ** |Metric|vLLM|SGLang| |:-|:-|:-| |Output throughput|**555.82 tok/s**|207.93 tok/s| |Mean TTFT|**549 ms**|1,558 ms| |Median TPOT|**7.06 ms**|18.84 ms| vLLM wins by 2.7x. SGLang is required `--quantization moe_…”
source: Reddit · u/NoVibeCoding · 2026-03-06
communityconfidence 75%
555.8tok/s — Qwen3-Coder on RTX 5090 via sglang AWQ
our signed data: RTX 5090 · Qwen3-Coder
“atency? # 1. Choosing the Framework **RTX 5090 — Qwen3-Coder-30B-A3B-Instruct-AWQ** |Metric|vLLM|SGLang| |:-|:-|:-| |Output throughput|**555.82 tok/s**|207.93 tok/s| |Mean TTFT|**549 ms**|1,558 ms| |Median TPOT|**7.06 ms**|18.84 ms| vLLM wins by 2.7x. SGLang is required `--qu…”
source: Reddit · u/NoVibeCoding · 2026-03-06
communityconfidence 75%
207.9tok/s — Qwen3-Coder on RTX 5090 via sglang AWQ
our signed data: RTX 5090 · Qwen3-Coder
“hoosing the Framework **RTX 5090 — Qwen3-Coder-30B-A3B-Instruct-AWQ** |Metric|vLLM|SGLang| |:-|:-|:-| |Output throughput|**555.82 tok/s**|207.93 tok/s| |Mean TTFT|**549 ms**|1,558 ms| |Median TPOT|**7.06 ms**|18.84 ms| vLLM wins by 2.7x. SGLang is required `--quantization moe_…”
source: Reddit · u/NoVibeCoding · 2026-03-06
communityconfidence 75%
555.8tok/s — Qwen3-Coder on RTX 5090 via sglang AWQ
our signed data: RTX 5090 · Qwen3-Coder
“atency? # 1. Choosing the Framework **RTX 5090 — Qwen3-Coder-30B-A3B-Instruct-AWQ** |Metric|vLLM|SGLang| |:-|:-|:-| |Output throughput|**555.82 tok/s**|207.93 tok/s| |Mean TTFT|**549 ms**|1,558 ms| |Median TPOT|**7.06 ms**|18.84 ms| vLLM wins by 2.7x. SGLang is required `--qu…”
source: Reddit · u/NoVibeCoding · 2026-03-06

See all 134 claims for RTX 5090 (32GB) →

Models measured on RTX 5090 (32GB)

Common questions about RTX 5090 (32GB)

Direct Q&A drawn from the runs above: fastest LLM, supported model classes, backend rankings, quantization guidance.

Read the RTX 5090 (32GB) FAQ →