2x RTX 3090 LLM benchmark
On the 2x RTX 3090, measured single-user chat decode speeds are: ~70B models at 19.1 tok/s (Q4_K_M, llama.cpp; Measured, 1 result, source run). Short-chat workload, local runtimes, dense models only; quantization, backend and settings vary between submissions, so check each row in the tables below.
Fastest single result in these submissions (any model size)
19.1 decode tok/s
Llama-3.3-70B-Instruct via llama.cpp (Q4_K_M). Workload: chat-short. Reported hardware: 2x RTX 3090 (24GB) + AMD EPYC 7H12 64-Core Processor (128c) + 1008GB. see full run
Llama-3.3-70B-Instruct
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | 2x RTX 3090 (24GB) + AMD EPYC 7H12 64-Core Processor (128c) + 1008GB | llama.cpp@0.7.0-dev (build 1, commit f74e2fa) | Q4_K_M | 19.14tok/s | 415.4tok/s | 325ms | r_n93ukk0mkmm |
| chat-long | not supplied | 2x RTX 3090 (24GB) + AMD EPYC 7H12 64-Core Processor (128c) + 1008GB | llama.cpp@0.7.0-dev (build 1, commit f74e2fa) | Q4_K_M | 18.69tok/s | 939.3tok/s | 3,366ms | r_n93ukk0mkmm |
| concurrent-decode | not supplied | 2x RTX 3090 (24GB) + AMD EPYC 7H12 64-Core Processor (128c) + 1008GB | llama.cpp@0.7.0-dev (build 1, commit f74e2fa) | Q4_K_M | 19.00tok/s | no data | no data | r_n93ukk0mkmm |
| agent-trace | not supplied | 2x RTX 3090 (24GB) + AMD EPYC 7H12 64-Core Processor (128c) + 1008GB | llama.cpp@0.7.0-dev (build 1, commit f74e2fa) | Q4_K_M | 18.85tok/s | 2,028.8tok/s | 1,037ms | r_n93ukk0mkmm |