RTX 5090 (32GB) LLM benchmark
The fastest LLM measured on the RTX 5090 (32GB) is stable-code-instruct-3b at 356.1 decode tok/s via llama.cpp (signed run). Across 97 reproducible runs on 19 models, this page lists decode tok/s, prefill, and TTFT for each, every number linking to the run it came from.
Fastest known config on RTX 5090 (32GB)
356.1 decode tok/s
stable-code-instruct-3b via llama.cpp. see full run
Coder-V2-Lite-Instruct
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 293.1tok/s | no data | 297ms | r_bfpto9so2o1 |
| chat-long | llama.cpp | - | 181.2tok/s | no data | 361ms | r_bfpto9so2o1 |
| concurrent-decode | llama.cpp | - | 268.4tok/s | no data | no data | r_bfpto9so2o1 |
| agent-trace | llama.cpp | - | 204.5tok/s | 24,978.7tok/s | 80.7ms | r_bfpto9so2o1 |
| chat-short | llama.cpp | - | 309.5tok/s | no data | 268ms | r_0_gs1rgl2fl |
gpt-oss-20b
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 318.4tok/s | no data | 173ms | r_b9ul-vxh9sc |
| chat-long | llama.cpp | - | 301.6tok/s | no data | 239ms | r_b9ul-vxh9sc |
| concurrent-decode | llama.cpp | - | 302.8tok/s | no data | no data | r_b9ul-vxh9sc |
| agent-trace | llama.cpp | - | 304.5tok/s | 26,907.5tok/s | 77.7ms | r_b9ul-vxh9sc |
| chat-short | llama.cpp | - | 69.42tok/s | no data | 333ms | r_r9h57uts9lr |
Yi-Coder-9B-Chat
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 66.79tok/s | no data | 317ms | r_zlh6az5q0o_ |
| chat-long | llama.cpp | - | 69.60tok/s | no data | 1,195ms | r_zlh6az5q0o_ |
| concurrent-decode | llama.cpp | - | 70.99tok/s | no data | no data | r_zlh6az5q0o_ |
| agent-trace | llama.cpp | - | 71.76tok/s | 4,478.0tok/s | 397ms | r_zlh6az5q0o_ |
| chat-short | llama.cpp | - | 199.3tok/s | no data | 36.2ms | r_u4iojm6-ekg |
stable-code-instruct-3b
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 341.9tok/s | no data | 175ms | r_q9f15lz6831 |
| chat-long | llama.cpp | - | 289.1tok/s | no data | 232ms | r_q9f15lz6831 |
| concurrent-decode | llama.cpp | - | 356.1tok/s | no data | no data | r_q9f15lz6831 |
| agent-trace | llama.cpp | - | 292.8tok/s | 44,387.5tok/s | 42.7ms | r_q9f15lz6831 |
| chat-short | llama.cpp | - | 331.0tok/s | no data | 36.7ms | r_8l57cim1i10 |
Qwen2.5-32B-Instruct
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 68.71tok/s | no data | 77.0ms | r_bjy5a5izxjc |
| chat-long | llama.cpp | - | 65.36tok/s | no data | 968ms | r_bjy5a5izxjc |
| concurrent-decode | llama.cpp | - | 68.60tok/s | no data | no data | r_bjy5a5izxjc |
| agent-trace | llama.cpp | - | 66.45tok/s | 10,407.9tok/s | 214ms | r_bjy5a5izxjc |
| chat-short | llama.cpp | - | 71.90tok/s | no data | 115ms | r_twfs86tf_xf |
Qwen2.5-14B-Instruct
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 130.3tok/s | no data | 48.6ms | r_xr4qdv1hgf2 |
| chat-long | llama.cpp | - | 122.7tok/s | no data | 489ms | r_xr4qdv1hgf2 |
| concurrent-decode | llama.cpp | - | 131.4tok/s | no data | no data | r_xr4qdv1hgf2 |
| agent-trace | llama.cpp | - | 122.8tok/s | 20,315.4tok/s | 111ms | r_xr4qdv1hgf2 |
| chat-short | llama.cpp | - | 133.3tok/s | no data | 55.9ms | r_tj9bu7gvnvh |
Llama-3.1-8B-Instruct
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 229.4tok/s | no data | 35.6ms | r_qr4srge34da |
| chat-long | llama.cpp | - | 211.4tok/s | no data | 266ms | r_qr4srge34da |
| concurrent-decode | llama.cpp | - | 223.0tok/s | no data | no data | r_qr4srge34da |
| agent-trace | llama.cpp | - | 210.3tok/s | 36,720.3tok/s | 60.8ms | r_qr4srge34da |
| chat-short | llama.cpp | - | 232.2tok/s | no data | 33.8ms | r_kfrkg-vn376 |
Qwen2.5-7B-Instruct
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 232.7tok/s | no data | 37.0ms | r_1shiviswt3d |
| chat-long | llama.cpp | - | 232.4tok/s | no data | 239ms | r_1shiviswt3d |
| concurrent-decode | llama.cpp | - | 240.7tok/s | no data | no data | r_1shiviswt3d |
| agent-trace | llama.cpp | - | 232.5tok/s | 38,007.7tok/s | 57.1ms | r_1shiviswt3d |
| chat-short | llama.cpp | - | 246.7tok/s | no data | 33.9ms | r_3yn-4321hp- |
Qwen3-32B
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 66.51tok/s | no data | 158ms | r_-txe_hiq44n |
| chat-long | llama.cpp | - | 62.30tok/s | no data | 1,067ms | r_-txe_hiq44n |
| concurrent-decode | llama.cpp | - | 66.64tok/s | no data | no data | r_-txe_hiq44n |
| agent-trace | llama.cpp | - | 64.34tok/s | 8,394.1tok/s | 259ms | r_-txe_hiq44n |
| chat-short | llama.cpp | - | 69.45tok/s | no data | 327ms | r_phvxm9dcak0 |
phi-4
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 139.1tok/s | no data | 51.1ms | r_k1u_k1j_1i2 |
| chat-long | llama.cpp | - | 133.7tok/s | no data | 433ms | r_k1u_k1j_1i2 |
| concurrent-decode | llama.cpp | - | 140.2tok/s | no data | no data | r_k1u_k1j_1i2 |
| agent-trace | llama.cpp | - | 124.8tok/s | 21,162.4tok/s | 102ms | r_k1u_k1j_1i2 |
| chat-short | llama.cpp | - | 140.9tok/s | no data | 56.7ms | r_e-k4aea8ipr |
gemma-2-9b-it
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 69.45tok/s | no data | 325ms | r_1_xl4zb5-xj |
| chat-long | llama.cpp | - | 69.42tok/s | no data | 1,044ms | r_1_xl4zb5-xj |
| concurrent-decode | llama.cpp | - | 68.57tok/s | no data | no data | r_1_xl4zb5-xj |
| agent-trace | llama.cpp | - | 68.46tok/s | 4,816.8tok/s | 420ms | r_1_xl4zb5-xj |
| chat-short | llama.cpp | - | 152.7tok/s | no data | 58.9ms | r__b_bzmmab_8 |
Qwen2.5-Coder-32B-Instruct
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 67.41tok/s | no data | 384ms | r_v983y0y3r2u |
| chat-long | llama.cpp | - | 69.15tok/s | no data | 1,251ms | r_v983y0y3r2u |
| concurrent-decode | llama.cpp | - | 70.96tok/s | no data | no data | r_v983y0y3r2u |
| agent-trace | llama.cpp | - | 68.56tok/s | 4,677.1tok/s | 414ms | r_v983y0y3r2u |
| chat-short | llama.cpp | - | 71.91tok/s | no data | 71.0ms | r_nkbs6d3-d21 |
Qwen2.5-Coder-14B-Instruct
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 140.0tok/s | no data | 42.1ms | r_opuj21f13-_ |
| chat-long | llama.cpp | - | 123.0tok/s | no data | 457ms | r_opuj21f13-_ |
| concurrent-decode | llama.cpp | - | 133.8tok/s | no data | no data | r_opuj21f13-_ |
| agent-trace | llama.cpp | - | 130.5tok/s | 21,985.9tok/s | 104ms | r_opuj21f13-_ |
| chat-short | llama.cpp | - | 136.0tok/s | no data | 51.3ms | r_p_f63tcgans |
Qwen2.5-Coder-7B-Instruct
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 255.3tok/s | no data | 35.4ms | r_2b9o6y_49mi |
| chat-long | llama.cpp | - | 244.0tok/s | no data | 228ms | r_2b9o6y_49mi |
| concurrent-decode | llama.cpp | - | 254.0tok/s | no data | no data | r_2b9o6y_49mi |
| agent-trace | llama.cpp | - | 229.9tok/s | 38,735.2tok/s | 57.4ms | r_2b9o6y_49mi |
| chat-short | llama.cpp | - | 244.9tok/s | no data | 36.1ms | r_mln72x5zbis |
Qwen3-Coder-30B-A3B-Instruct
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 68.41tok/s | no data | 334ms | r_pm_a1uf2ufc |
| chat-long | llama.cpp | - | 70.55tok/s | no data | 1,204ms | r_pm_a1uf2ufc |
| concurrent-decode | llama.cpp | - | 68.04tok/s | no data | no data | r_pm_a1uf2ufc |
| agent-trace | llama.cpp | - | 66.09tok/s | 4,816.7tok/s | 451ms | r_pm_a1uf2ufc |
| chat-short | llama.cpp | - | 259.9tok/s | no data | 218ms | r_c7qyvvmmsv1 |
Codestral-22B-v0.1
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 93.72tok/s | no data | 68.6ms | r_4q040m4scic |
| chat-long | llama.cpp | - | 87.36tok/s | no data | 849ms | r_4q040m4scic |
| concurrent-decode | llama.cpp | - | 100.3tok/s | no data | no data | r_4q040m4scic |
| agent-trace | llama.cpp | - | 93.44tok/s | 10,351.0tok/s | 191ms | r_4q040m4scic |
| chat-short | llama.cpp | - | 69.95tok/s | no data | 199ms | r_sqr8liqh4ii |
Qwen3.6-35B-A3B-Q4_K_M.gguf
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 223.3tok/s | no data | 263ms | r_dp6sr2_iwcf |
| chat-short | llama.cpp | - | 224.0tok/s | no data | 167ms | r_a56-wxl21lk |
| chat-short | llama.cpp | - | 216.5tok/s | no data | 185ms | r_k070lz99uzi |
| chat-short | llama.cpp | - | 221.6tok/s | no data | 207ms | r_dqelw_a8f2c |
gemma-4-31B-it-Q4_K_M.gguf
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 67.37tok/s | no data | 180ms | r_uut6m_v6ui9 |
| chat-short | llama.cpp | - | 67.27tok/s | no data | 344ms | r_23fireoga9y |
| chat-short | llama.cpp | - | 67.33tok/s | no data | 156ms | r_1ofpvf4m7p5 |
| chat-short | llama.cpp | - | 67.31tok/s | no data | 160ms | r_wwq2l7lmy6d |
Qwen3.6-27B-Q4_K_M.gguf
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 72.38tok/s | no data | 167ms | r_yluotk909p8 |
| chat-short | llama.cpp | - | 72.48tok/s | no data | 172ms | r_v2vrkr4uah1 |
| chat-short | llama.cpp | - | 69.56tok/s | no data | 227ms | r_u4wa_y_y3vt |
| chat-short | llama.cpp | - | 72.57tok/s | no data | 172ms | r_l-wg5a6o-vg |
| chat-short | llama.cpp | - | 69.56tok/s | no data | 174ms | r_w482rf72v6z |
| chat-short | llama.cpp | - | 72.32tok/s | no data | 290ms | r_ercjdbdw2gi |
| chat-short | llama.cpp | - | 69.60tok/s | no data | 382ms | r_aj808r0dw53 |
| chat-short | llama.cpp | - | 72.31tok/s | no data | 198ms | r_4upad7ubcpi |
| chat-short | llama.cpp | - | 68.00tok/s | no data | 241ms | r_qiq8q_cqfk5 |
Models measured on RTX 5090 (32GB)
- Coder-V2-Lite-Instruct benchmarks
- gpt-oss-20b benchmarks
- Yi-Coder-9B-Chat benchmarks
- stable-code-instruct-3b benchmarks
- Qwen2.5-32B-Instruct benchmarks
- Qwen2.5-14B-Instruct benchmarks
- Llama-3.1-8B-Instruct benchmarks
- Qwen2.5-7B-Instruct benchmarks
- Qwen3-32B benchmarks
- phi-4 benchmarks
- gemma-2-9b-it benchmarks
- Qwen2.5-Coder-32B-Instruct benchmarks
- Qwen2.5-Coder-14B-Instruct benchmarks
- Qwen2.5-Coder-7B-Instruct benchmarks
- Qwen3-Coder-30B-A3B-Instruct benchmarks
- Codestral-22B-v0.1 benchmarks
- Qwen3.6-35B-A3B-Q4_K_M.gguf benchmarks
- gemma-4-31B-it-Q4_K_M.gguf benchmarks
- Qwen3.6-27B-Q4_K_M.gguf benchmarks
Common questions about RTX 5090 (32GB)
Direct Q&A drawn from the runs above: fastest LLM, supported model classes, backend rankings, quantization guidance.