RTX 4090 (48GB) LLM benchmark
The fastest LLM measured on the RTX 4090 (48GB) is qwen2.5-coder at 161.1 decode tok/s via ollama (signed run). Across 16 reproducible runs on 3 models, this page lists decode tok/s, prefill, and TTFT for each, every number linking to the run it came from.
Fastest known config on RTX 4090 (48GB)
161.1 decode tok/s
qwen2.5-coder via ollama (Q4_K_M). see full run
qwen3.6
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_h_659oy695r |
| chat-long | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_h_659oy695r |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_h_659oy695r |
| agent-trace | ollama@0.31.1 | Q4_K_M | 44.18tok/s | 1,910.5tok/s | 1,808ms | r_h_659oy695r |
qwen2.5-coder
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.31.1 | Q4_K_M | 89.45tok/s | 466.6tok/s | 281ms | r_73tnfdueq2h |
| chat-long | ollama@0.31.1 | Q4_K_M | 83.84tok/s | 3,572.3tok/s | 887ms | r_73tnfdueq2h |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 88.63tok/s | no data | no data | r_73tnfdueq2h |
| agent-trace | ollama@0.31.1 | Q4_K_M | 86.21tok/s | 5,298.4tok/s | 395ms | r_73tnfdueq2h |
| chat-short | ollama@0.31.1 | Q4_K_M | 159.4tok/s | 452.0tok/s | 290ms | r_mv8n8k9wu1e |
| chat-long | ollama@0.31.1 | Q4_K_M | 154.7tok/s | 5,998.7tok/s | 528ms | r_mv8n8k9wu1e |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 161.1tok/s | no data | no data | r_mv8n8k9wu1e |
| agent-trace | ollama@0.31.1 | Q4_K_M | 158.1tok/s | 6,421.1tok/s | 320ms | r_mv8n8k9wu1e |
llama3.1
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.31.1 | Q4_K_M | 154.4tok/s | 328.0tok/s | 335ms | r_h1ub_1uxzdh |
| chat-long | ollama@0.31.1 | Q4_K_M | 146.3tok/s | 5,032.5tok/s | 623ms | r_h1ub_1uxzdh |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 154.2tok/s | no data | no data | r_h1ub_1uxzdh |
| agent-trace | ollama@0.31.1 | Q4_K_M | 149.4tok/s | 5,377.5tok/s | 387ms | r_h1ub_1uxzdh |
Models measured on RTX 4090 (48GB)
Common questions about RTX 4090 (48GB)
Direct Q&A drawn from the runs above: fastest LLM, supported model classes, backend rankings, quantization guidance.