Rtx A4000 LLM benchmark
The highest recorded decode rate in these Rtx A4000 submissions is from LFM2.5-24B-A2B at 140.0 decode tok/s via llama.cpp (source run). This page contains 4 workload rows from 1 submitted run across 1 model labels. Model sizes, workloads and reported hardware configurations differ; the highest rate is not a quality ranking or a speed guarantee.
Fastest single result in these submissions (any model size)
140.0 decode tok/s
LFM2.5-24B-A2B via llama.cpp (Q4_K_M). Workload: chat-short. Reported hardware: RTX A4000 (16GB) + AMD EPYC 7402P 24-Core Processor (24c) + 126GB. see full run
LFM2.5-24B-A2B
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX A4000 (16GB) + AMD EPYC 7402P 24-Core Processor (24c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit b76b133) | Q4_K_M | 140.0tok/s | 453.6tok/s | 251ms | r_j5tgvia-kjk |
| chat-long | not supplied | RTX A4000 (16GB) + AMD EPYC 7402P 24-Core Processor (24c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit b76b133) | Q4_K_M | 139.9tok/s | 2,684.4tok/s | 1,223ms | r_j5tgvia-kjk |
| concurrent-decode | not supplied | RTX A4000 (16GB) + AMD EPYC 7402P 24-Core Processor (24c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit b76b133) | Q4_K_M | 142.7tok/s | no data | no data | r_j5tgvia-kjk |
| agent-trace | not supplied | RTX A4000 (16GB) + AMD EPYC 7402P 24-Core Processor (24c) + 126GB | llama.cpp@0.7.0-dev (build 1, commit b76b133) | Q4_K_M | 139.4tok/s | 3,923.5tok/s | 391ms | r_j5tgvia-kjk |