RTX 3090 LLM benchmark
On the RTX 3090, measured single-user chat decode speeds are: ~7.6–8B models at 39.9 tok/s (Q4_K_M, ollama; Measured, middle of 2 results, source run), ~11.8–14.8B models at 78.2 tok/s (Q4_K_M, llama.cpp; Measured, middle of 3 results, source run) and ~27–32B models at 38.5 tok/s (Q4_K_M, llama.cpp; Measured, middle of 4 results, source run). Short-chat workload, local runtimes, dense models only; quantization, backend and settings vary between submissions, so check each row in the tables below.
Fastest single result in these submissions (any model size)
207.2 decode tok/s
gpt-oss-20b via llama.cpp (Q4_K_M). Workload: chat-short. Reported hardware: RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB. see full run
Qwen2.5-Coder-14B-Instruct
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 78.21tok/s | 1,693.2tok/s | 77.4ms | r_rkyf9wxzw9z |
| chat-long | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 74.31tok/s | 2,549.2tok/s | 1,243ms | r_rkyf9wxzw9z |
| concurrent-decode | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 77.11tok/s | no data | no data | r_rkyf9wxzw9z |
| agent-trace | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 75.86tok/s | 7,427.0tok/s | 273ms | r_rkyf9wxzw9z |
Qwen3-Coder-30B-A3B-Instruct
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 199.7tok/s | 612.5tok/s | 180ms | r_0dh1kzbv4h0 |
| chat-long | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 187.0tok/s | 3,971.3tok/s | 792ms | r_0dh1kzbv4h0 |
| concurrent-decode | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 198.0tok/s | no data | no data | r_0dh1kzbv4h0 |
| agent-trace | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 190.9tok/s | 10,182.2tok/s | 214ms | r_0dh1kzbv4h0 |
gpt-oss-20b
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 207.2tok/s | 658.3tok/s | 252ms | r_nakw1nq6clw |
| chat-long | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 199.8tok/s | 4,606.3tok/s | 692ms | r_nakw1nq6clw |
| concurrent-decode | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 205.1tok/s | no data | no data | r_nakw1nq6clw |
| agent-trace | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 202.2tok/s | 9,892.7tok/s | 224ms | r_nakw1nq6clw |
Qwen3.6-27B
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 42.73tok/s | 236.9tok/s | 473ms | r_qm1xyvjmx_r |
| chat-long | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 42.26tok/s | 1,321.5tok/s | 2,408ms | r_qm1xyvjmx_r |
| concurrent-decode | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 42.63tok/s | no data | no data | r_qm1xyvjmx_r |
| agent-trace | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 42.07tok/s | 2,369.6tok/s | 881ms | r_qm1xyvjmx_r |
Qwen3-32B
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 38.45tok/s | 589.5tok/s | 187ms | r_km62sybxga6 |
| chat-long | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 36.94tok/s | 1,218.2tok/s | 2,583ms | r_km62sybxga6 |
| concurrent-decode | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 38.05tok/s | no data | no data | r_km62sybxga6 |
| agent-trace | not supplied | RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 37.57tok/s | 3,287.7tok/s | 624ms | r_km62sybxga6 |
gemma3
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 11.8B | RTX 3090 (24GB) + AMD Ryzen Threadripper PRO 5975WX 32-Cores (32c) + 504GB | ollama@0.31.1 | Q4_K_M | 78.65tok/s | 26.02tok/s | 4,534ms | r_3gnxkzhhmyj |
| chat-long | 11.8B | RTX 3090 (24GB) + AMD Ryzen Threadripper PRO 5975WX 32-Cores (32c) + 504GB | ollama@0.31.1 | Q4_K_M | 74.24tok/s | 1,648.1tok/s | 1,933ms | r_3gnxkzhhmyj |
| concurrent-decode | 11.8B | RTX 3090 (24GB) + AMD Ryzen Threadripper PRO 5975WX 32-Cores (32c) + 504GB | ollama@0.31.1 | Q4_K_M | 76.01tok/s | no data | no data | r_3gnxkzhhmyj |
| agent-trace | 11.8B | RTX 3090 (24GB) + AMD Ryzen Threadripper PRO 5975WX 32-Cores (32c) + 504GB | ollama@0.31.1 | Q4_K_M | 74.48tok/s | 2,887.7tok/s | 808ms | r_3gnxkzhhmyj |
glm-4.7-flash
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 29.9B | RTX 3090 (24GB) + AMD Ryzen Threadripper PRO 5975WX 32-Cores (32c) + 504GB | ollama@0.31.1 | Q4_K_M | 134.6tok/s | 4.32tok/s | 24,309ms | r_kalwcyab-86 |
| chat-long | 29.9B | RTX 3090 (24GB) + AMD Ryzen Threadripper PRO 5975WX 32-Cores (32c) + 504GB | ollama@0.31.1 | Q4_K_M | 116.1tok/s | 2,090.1tok/s | 1,499ms | r_kalwcyab-86 |
| concurrent-decode | 29.9B | RTX 3090 (24GB) + AMD Ryzen Threadripper PRO 5975WX 32-Cores (32c) + 504GB | ollama@0.31.1 | Q4_K_M | 133.6tok/s | no data | no data | r_kalwcyab-86 |
| agent-trace | 29.9B | RTX 3090 (24GB) + AMD Ryzen Threadripper PRO 5975WX 32-Cores (32c) + 504GB | ollama@0.31.1 | Q4_K_M | 125.2tok/s | 2,901.4tok/s | 614ms | r_kalwcyab-86 |
Nemotron-3.5-Bolt-74B-A9B-24GB-ctx8k
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (64c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 80.32tok/s | 264.3tok/s | 454ms | r_w3lond-s75t |
| chat-long | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (64c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 80.36tok/s | 1,072.2tok/s | 3,030ms | r_w3lond-s75t |
| concurrent-decode | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (64c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 80.44tok/s | no data | no data | r_w3lond-s75t |
| agent-trace | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (64c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 79.25tok/s | 2,109.3tok/s | 944ms | r_w3lond-s75t |
Nemotron-3.5-Bolt-74B-A9B-24GB
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (64c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 87.75tok/s | 259.7tok/s | 462ms | r_-sifd5otbpy |
| chat-long | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (64c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 87.48tok/s | 1,183.4tok/s | 2,746ms | r_-sifd5otbpy |
| concurrent-decode | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (64c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 87.76tok/s | no data | no data | r_-sifd5otbpy |
| agent-trace | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (64c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 78.92tok/s | 2,095.0tok/s | 895ms | r_-sifd5otbpy |
Qwen3.8-27B
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (128c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit 8c2134a) | Q4_K_M | 37.65tok/s | 368.5tok/s | 418ms | r_ptct8o01o5l |
| chat-long | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (128c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit 8c2134a) | Q4_K_M | 37.25tok/s | 1,103.5tok/s | 2,921ms | r_ptct8o01o5l |
| concurrent-decode | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (128c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit 8c2134a) | Q4_K_M | 37.53tok/s | no data | no data | r_ptct8o01o5l |
| agent-trace | not supplied | RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (128c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit 8c2134a) | Q4_K_M | 36.75tok/s | 2,190.5tok/s | 968ms | r_ptct8o01o5l |
coder-v2
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 15.7B | RTX 3090 (24GB) + AMD EPYC 7702P 64-Core Processor (64c) + 252GB | ollama@0.31.1 | Q4_0 | 189.5tok/s | 159.2tok/s | 735ms | r_o2-1w665rtq |
| chat-long | 15.7B | RTX 3090 (24GB) + AMD EPYC 7702P 64-Core Processor (64c) + 252GB | ollama@0.31.1 | Q4_0 | 77.56tok/s | 3,907.5tok/s | 838ms | r_o2-1w665rtq |
| concurrent-decode | 15.7B | RTX 3090 (24GB) + AMD EPYC 7702P 64-Core Processor (64c) + 252GB | ollama@0.31.1 | Q4_0 | 155.6tok/s | no data | no data | r_o2-1w665rtq |
| agent-trace | 15.7B | RTX 3090 (24GB) + AMD EPYC 7702P 64-Core Processor (64c) + 252GB | ollama@0.31.1 | Q4_0 | 101.2tok/s | 6,933.1tok/s | 352ms | r_o2-1w665rtq |
qwen2.5-coder
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 14.8B | RTX 3090 (24GB) + AMD EPYC 7702P 64-Core Processor (64c) + 252GB | ollama@0.31.1 | Q4_K_M | 69.21tok/s | 402.0tok/s | 326ms | r_6ahy-dq0f_0 |
| chat-long | 14.8B | RTX 3090 (24GB) + AMD EPYC 7702P 64-Core Processor (64c) + 252GB | ollama@0.31.1 | Q4_K_M | 65.83tok/s | 2,070.6tok/s | 1,530ms | r_6ahy-dq0f_0 |
| concurrent-decode | 14.8B | RTX 3090 (24GB) + AMD EPYC 7702P 64-Core Processor (64c) + 252GB | ollama@0.31.1 | Q4_K_M | 67.64tok/s | no data | no data | r_6ahy-dq0f_0 |
| agent-trace | 14.8B | RTX 3090 (24GB) + AMD EPYC 7702P 64-Core Processor (64c) + 252GB | ollama@0.31.1 | Q4_K_M | 63.63tok/s | 3,518.8tok/s | 483ms | r_6ahy-dq0f_0 |
| chat-short | 7.6B | RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB | ollama@0.31.1 | Q4_K_M | 39.86tok/s | 2.83tok/s | 46,337ms | r_pb0jpnbujji |
| chat-long | 7.6B | RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB | ollama@0.31.1 | Q4_K_M | 135.0tok/s | 3,319.4tok/s | 954ms | r_pb0jpnbujji |
| concurrent-decode | 7.6B | RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB | ollama@0.31.1 | Q4_K_M | 139.2tok/s | no data | no data | r_pb0jpnbujji |
| agent-trace | 7.6B | RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB | ollama@0.31.1 | Q4_K_M | 134.2tok/s | 4,111.1tok/s | 487ms | r_pb0jpnbujji |
llama3.1
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 8.0B | RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB | ollama@0.31.1 | Q4_K_M | 136.2tok/s | 2.20tok/s | 49,960ms | r_9hurqggbshk |
| chat-long | 8.0B | RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB | ollama@0.31.1 | Q4_K_M | 127.5tok/s | 3,004.3tok/s | 1,044ms | r_9hurqggbshk |
| concurrent-decode | 8.0B | RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB | ollama@0.31.1 | Q4_K_M | 133.9tok/s | no data | no data | r_9hurqggbshk |
| agent-trace | 8.0B | RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB | ollama@0.31.1 | Q4_K_M | 127.5tok/s | 3,475.9tok/s | 527ms | r_9hurqggbshk |
Models measured on RTX 3090
- Qwen2.5-Coder-14B-Instruct benchmarks
- Qwen3-Coder-30B-A3B-Instruct benchmarks
- gpt-oss-20b benchmarks
- Qwen3.6-27B benchmarks
- Qwen3-32B benchmarks
- gemma3 benchmarks
- glm-4.7-flash benchmarks
- Nemotron-3.5-Bolt-74B-A9B-24GB-ctx8k benchmarks
- Nemotron-3.5-Bolt-74B-A9B-24GB benchmarks
- Qwen3.8-27B benchmarks
- coder-v2 benchmarks
- qwen2.5-coder benchmarks
- llama3.1 benchmarks