3x A100 LLM benchmark
The highest recorded decode rate in these 3x A100 submissions is from Muse-Spark-Open-816B-A42B-UD-00001-of-00005 at 25.5 decode tok/s via llama.cpp (source run). This page contains 4 workload rows from 1 submitted run across 1 model labels. Model sizes, workloads and reported hardware configurations differ; the highest rate is not a quality ranking or a speed guarantee.
Fastest single result in these submissions (any model size)
25.5 decode tok/s
Muse-Spark-Open-816B-A42B-UD-00001-of-00005 via llama.cpp (IQ1_S). Workload: chat-long. Reported hardware: 3x A100 (80GB) + AMD EPYC 7742 64-Core Processor (128c) + 1008GB. see full run
Muse-Spark-Open-816B-A42B-UD-00001-of-00005
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | 3x A100 (80GB) + AMD EPYC 7742 64-Core Processor (128c) + 1008GB | llama.cpp@0.7.0-dev (build 11675, commit 2851b3bcb) | IQ1_S | 25.20tok/s | 88.57tok/s | 2,732ms | r_fsd9qjmmna0 |
| chat-long | not supplied | 3x A100 (80GB) + AMD EPYC 7742 64-Core Processor (128c) + 1008GB | llama.cpp@0.7.0-dev (build 11675, commit 2851b3bcb) | IQ1_S | 25.50tok/s | 288.7tok/s | 11,179ms | r_fsd9qjmmna0 |
| concurrent-decode | not supplied | 3x A100 (80GB) + AMD EPYC 7742 64-Core Processor (128c) + 1008GB | llama.cpp@0.7.0-dev (build 11675, commit 2851b3bcb) | IQ1_S | 26.30tok/s | no data | no data | r_fsd9qjmmna0 |
| agent-trace | not supplied | 3x A100 (80GB) + AMD EPYC 7742 64-Core Processor (128c) + 1008GB | llama.cpp@0.7.0-dev (build 11675, commit 2851b3bcb) | IQ1_S | 22.79tok/s | 570.8tok/s | 3,896ms | r_fsd9qjmmna0 |