A40 LLM benchmark
On the A40, measured single-user chat decode speeds are: ~70B models at 13.0 tok/s (Q4_K_M, llama.cpp; Measured, 1 result, source run). Short-chat workload, local runtimes, dense models only; quantization, backend and settings vary between submissions, so check each row in the tables below.
Fastest single result in these submissions (any model size)
13.0 decode tok/s
Llama-3.3-70B-Instruct via llama.cpp (Q4_K_M). Workload: chat-short. Reported hardware: A40 (45GB) + Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz (48c) + 504GB. see full run
Llama-3.3-70B-Instruct
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | A40 (45GB) + Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz (48c) + 504GB | llama.cpp@0.7.0-dev (build 1, commit ceefd16) | Q4_K_M | 12.99tok/s | 401.0tok/s | 337ms | r_19f64yux97o |
| chat-long | not supplied | A40 (45GB) + Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz (48c) + 504GB | llama.cpp@0.7.0-dev (build 1, commit ceefd16) | Q4_K_M | 12.67tok/s | 551.4tok/s | 5,735ms | r_19f64yux97o |
| concurrent-decode | not supplied | A40 (45GB) + Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz (48c) + 504GB | llama.cpp@0.7.0-dev (build 1, commit ceefd16) | Q4_K_M | 12.85tok/s | no data | no data | r_19f64yux97o |
| agent-trace | not supplied | A40 (45GB) + Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz (48c) + 504GB | llama.cpp@0.7.0-dev (build 1, commit ceefd16) | Q4_K_M | 12.72tok/s | 1,574.5tok/s | 1,344ms | r_19f64yux97o |