Qwen3-8B-4bit
5 workload results across 2 hardware configurations.
Fastest config measured by llm-speed (@llmspeed)
155.0 decode tok/s
on RTX 4090 (24GB) via llama.cpp (Q4_K_M). see full run
Signed runs (5 workload results)
1 of 2 runs were measured by llm-speed (@llmspeed); the other one is community-submitted (signed on the submitter's own machine, not measured by llm-speed, and not used for llm-speed verdicts).
RTX 4090 (24GB)measured by llm-speed
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run | Source |
|---|---|---|---|---|---|---|---|
| chat-short | llama.cpp@0.7.0-dev (build 11803, commit a37cc4610) | Q4_K_M | 155.0tok/s | 2,155.2tok/s | 51.0ms | r_kiilk85gcx_ | Measured by llm-speed@llmspeed |
M3 Ultra (60-core GPU) + 96GB unified
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run | Source |
|---|---|---|---|---|---|---|---|
| chat-short | mlx@0.31.3 | 4BIT | 120.5tok/s | 446.1tok/s | 247ms | r_b1awwhkoo2v | Community-submitted |
| chat-long | mlx@0.31.3 | 4BIT | 103.3tok/s | 1,108.8tok/s | 2,838ms | r_b1awwhkoo2v | Community-submitted |
| concurrent-decode | mlx@0.31.3 | 4BIT | 114.3tok/s | no data | no data | r_b1awwhkoo2v | Community-submitted |
| agent-trace | mlx@0.31.3 | 4BIT | 109.4tok/s | 1,109.1tok/s | 1,885ms | r_b1awwhkoo2v | Community-submitted |
Community folklore
5 unverified claims extracted from Reddit/HN comments. Lower trust than signed runs above; every row links to the source.
- communityconfidence 75%
92.40tok/s — Qwen3-8B on RTX 4090 via llama.cpp Q8_0
our signed data: RTX 4090 · Qwen3-8B
“tok/s` generation at `128` output tokens - `Q8_0`: about `9975 tok/s` prompt processing at `512` tokens, `9955 tok/s` at `1024`, and about `92.4 tok/s` generation at `128` output tokens Hardware / runtime for those numbers: - `RTX 4090` - `Ryzen 9 7900X` - `llama.cpp` build com…”
- communityconfidence 75%
79.60tok/s — Qwen3-8B on M3 Max via mlx bf16
our signed data: M3 Max · Qwen3-8B
“Happy to coordinate if you open-source yours first — no point duplicating. EDIT: Benchmarks complete on M3 Max (128GB). I am getting 79.6 tok/s on Qwen3-8B-bf16 (3.41x speedup) with confirmed bit-for-bit parity against the baseline. For those interested in the MLX-specific …”
- communityconfidence 60%
73.00tok/s — Qwen3 8B on RTX 5060 Ti via llama.cpp
our signed data: RTX 5060 Ti · Qwen3 8B
“e even pytorch via mps???). A completely useless graph. I can also provide the same useless information: the Qwen3 8B model on my M1 gives 73 t/s and time to first token = 0.65s, which is faster than your rtx 5060ti. Now guess what type of M1 I have, what runtime I used, and wha…”
- communityconfidence 55%
60.00tok/s — Qwen3-8B via vllm fp4
our signed data: Qwen3-8B
“ance. So benchmarked two models for local inference: 1. Ollama serving qwen3:8b-q4\_K\_M = 70 t/s 2. VLLM serving nvidia/Qwen3-8B-NVFP4 = 60 t/s Both generated \~1000 tokens on a simple 50-token prompt. The token generation performance was reported via \`--verbose\` flag in ol…”
- communityconfidence 45%
711.0tok/s — Qwen3-8B Q8_0
our signed data: Qwen3-8B
“Run with --flash-attn ``` **Dense model (Qwen3-8B Q8_0) — prompt processing:** - ROCm default, no flash attn: **711 t/s** - ROCm + flash attn only: **~3,980 t/s** - **5.5× improvement from one flag** --- ## …”
Can Qwen3-8B-4bit run on your hardware?
Fastest decode speed on each GPU or Mac, from llm-speed's own runs and community-submitted runs. Each page lists every quant and runtime and labels which runs llm-speed measured.
Qwen3-8B-4bit on hardware
Check it for your setup
- Predict your exact setup: estimate whether Qwen3-8B-4bit fits your GPU or Mac at your quant and context, next to measured speeds where we have them.
- Request this pair: ask for a measured Qwen3-8B-4bit run on hardware we have not measured. Requests are ranked by votes.
- Full run table (2 runs): every Qwen3-8B-4bit result we have, with links to each source run.