Open data
The llm-speed dataset
Download 803 signed workload results from 356 public runs across 116 model labels and 19 reported hardware configurations: 419 measured by llm-speed and 384 community-submitted (not measured by llm-speed), marked in the source column. Each result links to its source run. CSV, JSON and focused studies with repeated measurements are available below under CC BY 4.0.
Current dataset version 2026-11-06 · exported 2026-11-06 · see the methodology for how each number is measured and signed.
Current dataset: version 2026-11-06
llm-speed signed LLM inference benchmarks (measured by llm-speed or community-submitted): 803 signed workload results from 356 public runs · 116 models · 19 hardware configurations · runtimes llama.cpp, mlx, ollama. One row per workload result; the source column says whether llm-speed measured it ('llm-speed') or it is community-submitted ('community'); every row links to its run page. Runs flagged for review are excluded. No submitter keys, fingerprints or handles are included.
Versions are dated and never overwritten; the newest version is listed in /data/open-dataset.json. License: CC BY 4.0.
Fields
- run_id
- Run ID; the run page is /r/<run_id>
- run_url
- Run page URL
- date
- Date the run was received (UTC, YYYY-MM-DD)
- hardware
- GPU (with its memory) or Apple Silicon as reported by the benchmark CLI; host CPU and RAM only for runs that kept work on the CPU (offload)
- model
- Model name as reported by the runtime
- model_size
- Parameter count as reported by the runtime, if any
- quant
- Quantization as reported by the runtime, if any
- runtime
- Inference runtime (backend), e.g. llama.cpp, ollama, mlx
- runtime_version
- Runtime version, if reported
- workload
- Suite workload, e.g. chat-short, agent-trace
- suite_version
- Benchmark suite version
- decode_tps
- Decode speed recorded by the signed run, tokens/second
- prefill_tps
- Prompt-processing speed recorded by the signed run, tokens/second
- ttft_ms
- Time to first token recorded by the signed run, ms
- prompt_tokens
- Prompt tokens
- output_tokens
- Generated tokens
- batch_size
- Concurrent sequences
- source
- 'llm-speed' when llm-speed measured the run (listed in https://api.llm-speed.com/v1/run-sources), else 'community' (community-submitted, not measured by llm-speed)
How to cite
llm-speed. Signed LLM inference benchmarks (measured by llm-speed or community-submitted), version 2026-11-06. https://llm-speed.com/data. CC BY 4.0.Focused benchmark studies
Qwen3.8-27B and Gemma 4 12B RTX 5090 chat benchmarks — September 5, 2026
Twelve workload measurements from six accepted runs on one RTX 5090: three repetitions each of Qwen3.8-27B Q4_K_M and Gemma 4 12B QAT Q4_0, covering short and longer chat prompts. Includes decode speed, first-token latency, actual token lengths, pinned artifacts and configuration limits; no coding-quality evaluation.
Published September 12, 2026 · measured September 5 · CC BY 4.0. This study is separate from the July bulk snapshot below.
All 6 runs were measured by llm-speed (@llmspeed).
Gemma 4 12B IT QAT long-context benchmarks on RTX 5090 — October 22, 2026
Nine measurements from three runs of the long-context-decay workload (context targets of about 32K, 64K and 128K tokens at a 131,072-token configured context) with Gemma 4 12B IT QAT Q4_0 on one RTX 5090, llama.cpp CUDA 9725a31. Includes decode speed, prefill speed, time to first token and actual prompt lengths. It reruns the September 6 community-submitted runs with the same commit, model file, context and workload.
Published and measured October 22, 2026 · CC BY 4.0. The rows are also in open dataset version 2026-10-22.3 and later.
All 3 runs were measured by llm-speed (@llmspeed).
Archived July 8 summary snapshot
The July snapshot is also published as a HuggingFace dataset (with a live Dataset Viewer) and archived on Zenodo with a citable DOI, for notebooks, training pipelines, and papers.
Prefer more recent submissions? The JSON API returns a bounded list of run summaries (50 by default); it is not a full-corpus download. Per-run detail is at api.llm-speed.com/v1/results/{id}. Every run also has a citable page at llm-speed.com/r/{id}.
Reuse it (attribution required)
The data is licensed Creative Commons Attribution 4.0. Use it anywhere, including commercially, as long as you credit llm-speed with a link back. Copy one of these:
Data from llm-speed (https://llm-speed.com/data), CC BY 4.0.<a href="https://llm-speed.com/data">Benchmark data by llm-speed</a>, CC BY 4.0.Building a tool on this data (a VRAM calculator, a price-per-token comparison, a hardware guide)? That is exactly what the license is for. A link back is all we ask.
Citing it in a post or paper? Use:
llm-speed. llm-speed: signed LLM inference-speed benchmarks. July 8, 2026 snapshot. Zenodo (2026). https://doi.org/10.5281/zenodo.21254813. CC BY 4.0.How to interpret the data
These are submitted measurements, with varying artifacts, workloads and runtime settings. A signature preserves submitted provenance; it does not independently verify hardware, answer quality or exact reproducibility. The bulk summaries omit signatures and per-workload detail. Open a source permalink before comparing rates, and preserve missing fields instead of inferring a configuration. The focused study above documents its exact cohort and limitations separately.
What is in each row of the July snapshot
idandpermalink:the source run and its citable pagetop_model_name,top_backend,top_workload:what was runaccelerator_summary:the GPU or Apple Silicon plus hosttop_decode_tps:the headline decode tokens per secondreceived_at,suite_version:when it landed and under which suite
The numbers are measured, not modeled. If a run looks wrong, open its permalink and check the raw workload results. Browse the same data as a leaderboard on the cheatsheet.