Cheatsheet

Local LLM speed cheatsheet: decode tok/s by GPU & Mac

Name: LLM speed cheatsheet — every model x every rig
Creator: llm-speed
License: https://creativecommons.org/licenses/by/4.0/

Best decode tokens-per-second per (model × hardware) tuple measured by llm-speed under suite-v1. Numbers are wall-clock, batch size 1 unless the workload says otherwise. Each row links to the canonical run page; cite as llm-speed.com/r/<id>.

157 (model × hardware) combinations · sorted by decode tok/s · suite-v1

Long documents and bulk extraction

For long inputs with short answers, compare prefill, time to first token and total request time at your input length, cache state and concurrency. Check extraction accuracy too. This table selects the fastest decode row per model and hardware pair; its prefill and TTFT columns describe that row, not the best prefill speed or lowest latency.

Start with the prefill vs decode guide or the measured Gemma 4 long-context study to see how input length changes the wait before streaming.

Model	Quant	Hardware	Backend	Workload	decode tok/s	prefill tok/s	TTFT (ms)	Run
llama3.2	Q8_0	RTX 5880 (48GB)	ollama@0.34.2	chat-short	466.9	44.3	2820.6	r_b948ehgi8g4
mlx-community/Qwen3-0.6B-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	429.2	1185.2	92.8	r_6rgao6ao735
stable-code-instruct-3b	-	RTX 5090 (32GB)	llama.cpp	concurrent-decode	356.1	—	—	r_q9f15lz6831
gpt-oss-20b	Q4_K_M	RTX 5090 (32GB)	llama.cpp@0.7.0-dev (build 1, commit 1268eda)	chat-short	324.2	586.8	282.9	r_dwp5fgdapmw
LFM2.5-24B-A2B	Q4_K_M	RTX 5090 (32GB)	llama.cpp@0.6.0-dev (build 1, commit a0e6dc7)	concurrent-decode	321.3	—	—	r_c8k6fleym09
DeepSeek-Coder-V2-Lite-Instruct	-	RTX 5090 (32GB)	llama.cpp	chat-short	309.5	—	267.6	r_0_gs1rgl2fl
Qwen3-Coder-30B-A3B-Instruct	Q4_K_M	RTX 5090 (32GB)	llama.cpp@0.7.0-dev (build 1, commit 1268eda)	chat-short	304.9	514.5	213.8	r_vsype4pm1ko
mlx-community-Qwen2.5-0.5B-Instruct-4bit	4bit	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	286.5	656.2	199.6	r_akcbpx5vcqa
Qwen2.5-Coder-7B-Instruct	-	RTX 5090 (32GB)	llama.cpp	chat-short	255.3	—	35.4	r_2b9o6y_49mi
Qwen2.5-7B-Instruct	-	RTX 5090 (32GB)	llama.cpp	chat-short	246.7	—	33.9	r_3yn-4321hp-
gpt-oss-20b	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.6.0-dev (build 1, commit df18bee)	chat-short	242.6	593.8	279.5	r_t-a9v_ipdit
LFM2.5-24B-A2B	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.6.0-dev (build 1, commit a0e6dc7)	concurrent-decode	236.8	—	—	r_bqmpv9kncyd
Qwen3-Coder-30B-A3B-Instruct	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.6.0-dev (build 1, commit df18bee)	chat-short	232.5	831.9	132.2	r_1k8942tw3zf
Llama-3.1-8B-Instruct	-	RTX 5090 (32GB)	llama.cpp	chat-short	232.2	—	33.8	r_kfrkg-vn376
Qwen3.6-35B-A3B-Q4_K_M.gguf	-	RTX 5090 (32GB)	llama.cpp	chat-short	224.0	—	169.5	r_g1aw3mizd32
Sage-A1.5-ctx48k	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	concurrent-decode	217.2	—	—	r_izwoxrsouc4
Sage-A1.5-ctx8k	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	concurrent-decode	216.9	—	—	r_4x-4g-xhhgk
Sage-A1.5-ctx128k	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	216.8	1023.9	109.4	r_-i4k-ieitrk
Sage-A1.5-ctx32k	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	216.8	1023.6	109.4	r_9uktwjj22y-
Sage-A1.5-ctx64k	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	216.6	1042.3	107.5	r_vusd3gw5tpg
Sage-A1.5-ctx96k	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	concurrent-decode	216.6	—	—	r_5nh2jxjbzur
Sage-A1.5-ctx16k	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	216.6	991.6	113.0	r_anosq1xr6tc
gpt-oss-20b	Q4_K_M	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 1, commit ea0c4c2)	chat-short	207.2	658.3	252.1	r_nakw1nq6clw
Qwen3-Coder-30B-A3B-Instruct	Q4_K_M	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 1, commit ea0c4c2)	chat-short	199.7	612.5	179.6	r_0dh1kzbv4h0
Yi-Coder-9B-Chat	-	RTX 5090 (32GB)	llama.cpp	chat-short	199.3	—	36.2	r_u4iojm6-ekg
gemma3	Q4_K_M	RTX 4090 (24GB)	ollama@0.31.1	chat-short	195.0	167.2	705.6	r_dlanfbgym0h
mlx-community/stable-code-instruct-3b-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	192.5	560.7	226.5	r_y2_5y8oo97d
deepseek-coder-v2	Q4_0	RTX 3090 (24GB)	ollama@0.31.1	chat-short	189.5	159.2	734.9	r_o2-1w665rtq
mlx-community/Qwen3-4B-Instruct-2507-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	182.7	636.6	172.8	r_p2rpb_0iyjj
qwen3-coder	Q4_K_M	RTX 4090 (24GB)	ollama@0.31.1	concurrent-decode	179.9	—	—	r_wjq32z47vlp
Flint-40B-A2.5B-NVFP4	-	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	179.4	423.1	231.6	r_0i0a9uq6fdx
Flint-40B-A2.5B-UD-ctx16k	Q3_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	169.2	442.0	221.7	r_sw92j6kg7js
mlx-community/DeepSeek-Coder-V2-Lite-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	168.3	291.5	449.5	r_l_v1-zq_qaz
Flint-40B-A2.5B-UD-ctx8k	Q3_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	168.0	488.4	200.6	r_r7gzl4lmmdc
glm-4.7-flash	Q4_K_M	RTX 4090 (24GB)	ollama@0.31.1	chat-short	164.9	3.7	28105.7	r_ted0u8uyre0
Qwen3.6-35B-A3B	Q4_K_M	RTX 4090 (24GB)	llama.cpp@8931 (9725a313b)	concurrent-decode	163.5	—	—	r_m6j1zxwvp-f
qwen2.5-coder	Q4_K_M	RTX 4090 (48GB)	ollama@0.31.1	concurrent-decode	161.1	—	—	r_mv8n8k9wu1e
Meta-Llama-3.1-8B-Instruct	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	160.8	4283.1	31.5	r_32ipo1hbbwl
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-ctx32k	-	RTX 5090 (32GB)	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	157.7	—	—	r_ymx58emedxd
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB	-	RTX 5090 (32GB)	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	156.0	—	—	r_z-l1yelv_67
DeepSeek-R1-0528-Qwen3-8B	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	155.0	2106.7	49.8	r_ll6l5npoik8
Qwen3-8B	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	155.0	2155.2	51.0	r_kiilk85gcx_
llama3.1	Q4_K_M	RTX 4090 (48GB)	ollama@0.31.1	chat-short	154.4	328.0	335.4	r_h1ub_1uxzdh
gemma-2-9b-it	-	RTX 5090 (32GB)	llama.cpp	chat-short	152.7	—	58.9	r__b_bzmmab_8
mlx-community/gpt-oss-20b-MXFP4-Q4	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	152.7	239.9	692.1	r_3ijun8ltjnb
LFM2.5-24B-A2B	Q4_K_M	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 1, commit b76b133)	concurrent-decode	142.7	—	—	r_j5tgvia-kjk
gemma-4-12b-it-qat	Q4_0	RTX 5090 (32GB)	llama.cpp@1 (9725a31)	long-context-decay	142.6	7115.8	3354.8	r_-lh-vczf7ik
gpt-oss	MXFP4	RTX 4090 (24GB)	ollama@0.31.1	chat-long	141.7	631.5	5048.2	r_iu2sfa9ykvw
phi-4	-	RTX 5090 (32GB)	llama.cpp	chat-short	140.9	—	56.7	r_e-k4aea8ipr
Qwen2.5-Coder-14B-Instruct	-	RTX 5090 (32GB)	llama.cpp	chat-short	140.0	—	42.1	r_opuj21f13-_
mlx-community-Qwen2.5-7B-Instruct-4bit	4bit	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	139.6	190.0	689.4	r_5r6rhiynenc
qwen2.5-coder	Q4_K_M	RTX 3090 (24GB)	ollama@0.31.1	concurrent-decode	139.2	—	—	r_pb0jpnbujji
mlx-community/Qwen2.5-Coder-7B-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	138.6	539.6	242.8	r_uoehjq0nvc0
llama3.1	Q4_K_M	RTX 4090 (24GB)	ollama@0.31.1	chat-short	137.3	269.1	408.8	r_2np7abzjobe
qwen2.5-coder	Q4_K_M	RTX 4090 (24GB)	ollama@0.31.1	concurrent-decode	137.3	—	—	r_nrpc8-onxu2
llama3.1	Q4_K_M	RTX 3090 (24GB)	ollama@0.31.1	chat-short	136.2	2.2	49960.0	r_9hurqggbshk
glm-4.7-flash	Q4_K_M	RTX 3090 (24GB)	ollama@0.31.1	chat-short	134.6	4.3	24308.6	r_kalwcyab-86
deepseek-r1	Q4_K_M	RTX 4090 (24GB)	ollama@0.31.1	agent-trace	133.8	8852.4	366.2	r_fg77v2hhohb
Qwen2.5-14B-Instruct	-	RTX 5090 (32GB)	llama.cpp	chat-short	133.3	—	55.9	r_tj9bu7gvnvh
Flint-40B-A2.5B-UD-ctx8k	Q3_K_XL	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	130.8	365.9	267.8	r_-yeg4qnhjp7
mlx-community/Llama-3.1-8B-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	130.2	400.3	339.7	r_v2pbc0rq2l4
Flint-40B-A2.5B-UD-ctx16k	Q3_K_XL	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	128.9	371.7	263.6	r_v-sdiljpm7i
mlx-community/Qwen3-8B-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	120.5	446.1	246.6	r_b1awwhkoo2v
lmstudio-community/DeepSeek-R1-0528-Qwen3-8B-MLX-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	116.8	424.5	247.4	r_yy_6jfx70jq
qwen3-coder-bench-32k	Q4_K_M	M4 Max (40-core GPU) + 128GB unified	ollama@0.30.11	chat-short	113.3	84.1	1307.7	r_roktphpc--8
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	112.5	308.7	356.3	r_o6wbmfegmde
mlx-community/granite-8b-code-instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	112.3	525.7	220.7	r_bue3bee0gw7
mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	112.2	204.0	539.3	r_fpsca03u2o_
qwen3-coder	Q4_K_M	M4 Max (40-core GPU) + 128GB unified	ollama@0.30.11	concurrent-decode	109.9	—	—	r_r0di2hkku1h
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB	-	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	chat-short	107.3	255.8	469.1	r_vkjsinuhlyl
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-ctx8k	-	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	chat-short	107.0	253.3	473.7	r_iu0s18cs9d8
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-ctx16k	-	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	105.8	—	—	r_qibhq5bkmgq
mlx-community/Yi-Coder-9B-Chat-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	103.5	390.9	307.0	r_3hvui9a1yuc
llama3.2	Q8_0	M1 Pro (16-core GPU) + 16GB unified	ollama@0.32.14	chat-short	103.1	8.8	14246.7	r_w-p7v--eync
Codestral-22B-v0.1	-	RTX 5090 (32GB)	llama.cpp	concurrent-decode	100.3	—	—	r_4q040m4scic
gemma-4-12b-it-qat	Q4_0	RTX 4090 (24GB)	llama.cpp@8931 (9725a313b)	chat-short	97.7	1109.0	111.8	r_jfh0qkngw76
Flint-40B-A2.5B-UD-ctx8k	IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	92.9	331.7	295.5	r_e8r7fxw1ke9
Qwen2.5-Coder-14B-Instruct	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.6.0-dev (build 1, commit df18bee)	chat-short	89.9	2292.6	57.1	r_eqnsqmgz29a
mlx-community/gemma-2-9b-it-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	89.5	170.7	691.4	r_iz137eqvuzy
Flint-40B-A2.5B-UD-ctx16k	IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	89.4	327.0	299.7	r_tp8imqk28q5
qwen3.6-agent-q6k-ctx128k	Q6_K	M5 Max (40-core GPU) + 48GB unified	ollama@0.32.1	chat-short	88.3	16.2	6915.3	r_udwgba7udqu
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB	-	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	87.8	—	—	r_-sifd5otbpy
Qwen3.8-27B-mtp-on	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.6.0-dev (build 1, commit df18bee)	concurrent-decode	87.0	—	—	r_7j8rpk9mwhg
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-ctx8k	-	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	80.4	—	—	r_w3lond-s75t
lmstudio-community-Qwen3-Next-80B-A3B-Instruct-MLX-4bit	4bit	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	80.3	24.5	4492.5	r_1pl79r50ofy
gemma3	Q4_K_M	RTX 3090 (24GB)	ollama@0.31.1	chat-short	78.7	26.0	4534.1	r_3gnxkzhhmyj
Qwen2.5-Coder-14B-Instruct	Q4_K_M	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 1, commit ea0c4c2)	chat-short	78.2	1693.2	77.4	r_rkyf9wxzw9z
Qwen3.8-27B	Q4_K_M	RTX 5090 (32GB)	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	75.4	—	—	r_qfrb2zmyk0k
Qwen3.6-27B	Q4_K_M	RTX 5090 (32GB)	llama.cpp@0.7.0-dev (build 1, commit 1268eda)	concurrent-decode	75.1	—	—	r_pob-co51bd-
mlx-community/phi-4-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	74.4	255.0	443.2	r_sqzp-0rdez-
Qwen3.6-27B-Q4_K_M.gguf	-	RTX 5090 (32GB)	llama.cpp	chat-short	74.0	—	190.4	r_f7ulllqu4vk
Qwen2.5-Coder-32B-Instruct	-	RTX 5090 (32GB)	llama.cpp	chat-short	71.9	—	71.0	r_nkbs6d3-d21
Qwen2.5-32B-Instruct	-	RTX 5090 (32GB)	llama.cpp	chat-short	71.9	—	115.2	r_twfs86tf_xf
mlx-community-Qwen2.5-14B-Instruct-4bit	4bit	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	70.9	301.7	434.1	r_v4bq1sviz4o
mlx-community/Qwen2.5-Coder-14B-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	70.5	306.1	427.9	r_l36cijqxq4t
Qwen3-32B	Q4_K_M	RTX 5090 (32GB)	llama.cpp@0.7.0-dev (build 1, commit 1268eda)	chat-short	69.8	1047.0	105.1	r_vhbqyt1f_xr
gemma-4-31B-it-Q4_K_M.gguf	-	RTX 5090 (32GB)	llama.cpp	chat-short	69.7	—	174.8	r_pa7nuerd9tl
mlx-community/starcoder2-15b-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	64.2	220.5	489.8	r_wsxml_39dh_
Muse-Glimmer-30B-ctx8k	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	53.7	655.3	235.0	r_21z5pn8262x
Muse-Glimmer-30B-ctx32k	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	53.7	655.0	235.1	r_sz089qf84ge
Muse-Glimmer-30B-ctx128k	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	53.7	651.7	236.3	r_jn2059subhm
Muse-Glimmer-30B-ctx48k	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	53.7	656.9	234.5	r_j8qmkq2kd8l
Muse-Glimmer-30B-ctx96k	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	53.7	656.2	234.7	r_xva_h_3wgzn
Muse-Glimmer-30B-ctx64k	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	53.6	679.9	226.5	r_d6vbcmoirgr
Muse-Glimmer-30B-ctx96k-kvq8	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	52.7	620.6	248.2	r_j_abawmxlle
Muse-Glimmer-30B-ctx64k-kvq8	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	52.7	685.5	224.6	r_d1mp5onfwe8
Muse-Glimmer-30B-ctx128k-kvq8	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	52.7	614.6	250.6	r_8dtxula7h35
Muse-Glimmer-30B-ctx32k-kvq8	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	52.7	618.7	248.9	r_g5vn9b96z6e
Muse-Glimmer-30B-ctx48k-kvq8	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	52.7	614.6	250.6	r_7srd_wfvix-
Muse-Glimmer-30B-ctx8k-kvq8	UD-Q4_K_XL	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	52.7	617.4	249.4	r_sbcga8o1-af
mlx-community/Codestral-22B-v0.1-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	47.5	205.7	559.1	r_79dvtag5fd_
Qwen3.6-27B	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.6.0-dev (build 1, commit 3fb4767)	concurrent-decode	47.2	—	—	r_d_wda_rw62f
qwen3.6	Q4_K_M	RTX 4090 (48GB)	ollama@0.31.1	agent-trace	44.2	1910.5	1807.9	r_h_659oy695r
Qwen3.8-27B-ctx8k	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	44.2	691.5	222.7	r_tt9io-8lu9p
Qwen3-32B	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.6.0-dev (build 1, commit df18bee)	chat-short	43.5	778.6	141.3	r_ayy9rjgha5r
Qwen3.8-27B-ctx4k	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	43.4	448.3	343.6	r_7cap38a021-
Qwen3.8-27B-ctx16k	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	43.4	472.6	325.9	r_cthpfm82n1r
Qwen3.8-27B	Q4_K_M	RTX 4090 (24GB)	llama.cpp@0.6.0-dev (build 1, commit df18bee)	chat-short	43.1	428.6	359.3	r_11kn1epxewh
Qwen3.6-27B	Q4_K_M	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 1, commit ea0c4c2)	chat-short	42.7	236.9	472.7	r_qm1xyvjmx_r
Flint-40B-A2.5B-UD-cpumoe6-ctx8k	Q4_K_XL	RTX 4090 (24GB) + AMD EPYC 7452 32-Core Processor (32c) + 504GB	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	concurrent-decode	42.7	—	—	r_wap61ewkpzm
Qwen3.8-27B-ctx8k	Q4_K_M	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	38.8	479.1	321.5	r_w40_vv7f77w
Qwen3.8-27B-ctx4k	Q4_K_M	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	38.6	482.3	319.3	r_mmtzel_i367
Qwen3.8-27B-ctx16k	Q4_K_M	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	38.6	482.7	319.0	r_gscy1f7ey_k
Qwen3-32B	Q4_K_M	RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 1, commit ea0c4c2)	chat-short	38.5	589.5	186.6	r_km62sybxga6
Qwen3.8-27B	Q4_K_M	RTX 3090 (24GB)	llama.cpp@0.6.0-dev (build 1, commit 8c2134a)	chat-short	37.6	368.5	417.9	r_ptct8o01o5l
mlx-community-Qwen2.5-32B-Instruct-4bit	4bit	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	34.6	141.9	922.9	r_njgxtgyym1e
mlx-community/Qwen2.5-Coder-32B-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	34.5	144.1	909.4	r_721b4bls_oq
mlx-community/Qwen3-32B-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	34.4	95.1	1156.4	r_anmmc80-aoq
mlx-community-Qwen2.5-7B-Instruct-4bit	4bit	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	30.5	161.9	809.1	r_llzv_g-ymaf
mlx-community/Llama-3.1-8B-Instruct-4bit	-	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	29.2	203.3	669.0	r_h0-use1ypnb
Muse-Spark-Open-816B-A42B-UD-00001-of-00005	IQ1_S	3x A100 (80GB)	llama.cpp@0.7.0-dev (build 11675, commit 2851b3bcb)	concurrent-decode	26.3	—	—	r_fsd9qjmmna0
llama3.2	Q8_0	AMD64 Family 23 Model 104 Stepping 1, AuthenticAMD (6c) + 15GB	ollama@0.34.0	chat-short	23.9	39.4	3171.6	r_b1rn5gi6b3b
Muse-Glimmer-30B-ctx8k	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	23.3	265.1	580.8	r_u_kr0v-wwde
mlx-community/gemma-2-9b-it-4bit	-	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	22.0	102.5	1151.3	r_q7t7b7dcuz5
Muse-Glimmer-30B-ctx128k	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.9	255.0	603.9	r_1lyb_mqmdik
Muse-Glimmer-30B-ctx96k-kvq8	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.8	264.1	583.1	r_0o1xfrrn-ch
Muse-Glimmer-30B-ctx48k-kvq8	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.7	273.7	562.7	r_x8mqtlf166j
Muse-Glimmer-30B-ctx16k-kvq8	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.7	253.3	607.9	r_dldd3d54262
Muse-Glimmer-30B-ctx8k-kvq8	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.7	271.4	567.4	r_b8y4oh7e0-z
Muse-Glimmer-30B-ctx128k-kvq8	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.7	246.1	625.8	r_crqe50ubylk
Muse-Glimmer-30B-ctx32k-kvq8	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.6	272.3	565.6	r_zoeatakzums
Muse-Glimmer-30B-ctx96k	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.6	265.7	579.7	r_pf4ye8_4xmc
Muse-Glimmer-30B-ctx64k-kvq8	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.5	266.6	577.7	r_aufjud7duqr
Muse-Glimmer-30B-ctx64k	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.5	274.6	560.8	r_j9yaui8ovvy
Muse-Glimmer-30B-ctx32k	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.5	269.7	571.0	r_ntsos2cxwja
Muse-Glimmer-30B-ctx48k	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.5	271.6	566.9	r_y8qy4yd5p7i
Muse-Glimmer-30B-ctx16k	UD-IQ3_XXS	RTX A4000 (16GB)	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	21.4	256.7	600.0	r_y9y09s77lfa
Llama-3.3-70B-Instruct	Q4_K_M	2x RTX 3090 (24GB)	llama.cpp@0.7.0-dev (build 1, commit f74e2fa)	chat-short	19.1	415.4	325.0	r_n93ukk0mkmm
Flint-40B-A2.5B-UD-cpumoe6-ctx8k	Q4_K_XL	RTX 3090 (24GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-long	17.4	1316.2	2290.0	r_klxxsjjp2m-
mlx-community/llama-3.3-70b-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	16.8	25.1	5419.9	r_sx3a4y9n-m4
mlx-community/qwen2.5-72b-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	16.3	23.2	5635.3	r_5c80gthqlh6
Flint-40B-A2.5B-UD-cpumoe12-ctx8k	Q3_K_XL	RTX A4000 (16GB) + AMD EPYC 7453 28-Core Processor (56c) + 504GB	llama.cpp@0.7.0-dev (build 11803, commit a37cc4610)	chat-short	15.7	190.0	515.8	r_acy-a8qic5j
mlx-community/phi-4-4bit	-	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	15.6	120.7	936.0	r_-w8hnn61va_
mlx-community-Qwen2.5-14B-Instruct-4bit	4bit	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	14.4	117.9	1111.5	r_ia73dzeue0b
Llama-3.3-70B-Instruct	Q4_K_M	A40 (45GB)	llama.cpp@0.7.0-dev (build 1, commit ceefd16)	chat-short	13.0	401.0	336.7	r_19f64yux97o
mlx-community/Qwen3-32B-4bit	-	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	7.2	25.7	4287.9	r_pnrrpcdqfo4
mlx-community/Qwen2.5-32B-Instruct-4bit	-	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	7.1	33.4	3921.1	r_f4x3xaan2ay

A40 (45GB): 48GB, ~45 GiB usable. The label shows what its driver reports as usable, not the 48GB spec.

How to read this table

Each row is the highest decode tokens-per-second measured for a unique (model, hardware) pair. When more than one workload was run, we pick the workload that produced the headline number. Lower-tps duplicates from the same machine are not shown; click through to the run page for the full per-workload breakdown.

decode tok/s is wall-clock streaming throughput (memory-bandwidth-bound). prefill tok/s is prompt ingestion throughput (compute-bound). TTFT is time-to-first-token in milliseconds. See /glossary for definitions and /methodology for the workload spec.