Can an RTX A4000 run Flint 40B-A2.5B?

Only at UD-IQ3_XXS: an RTX A4000 runs Flint 40B-A2.5B at 92.9 tok/s at UD-IQ3_XXS, measured on llama.cpp with 8K context in October 2026.

3 signed runs (2 default settings, 1 CPU offload) on llm-speed. Single-request decode speed in tokens per second; the short-chat workload unless noted. All runs were measured by llm-speed (@llmspeed). Runs measured by llm-speed are credited to @llmspeed. TTFT includes reasoning tokens (time to the first token, not to the first answer); chat-short prefill is not shown because its short prompt makes it unrepresentative, so see chat-long for prefill.

Measured tok/s by quant and runtime

Quant	Runtime	Build	Decode tok/s (best run)	Workload	Runs	Source
UD-IQ3_XXS	llama.cpp	commit a37cc4610	92.9	chat-short	2	Measured by llm-speed@llmspeed
UD-Q3_K_XL with CPU offload	llama.cpp	commit a37cc4610	15.7	chat-short	1	Measured by llm-speed@llmspeed

Fastest quant that fits: UD-IQ3_XXS on llama.cpp, 92.9 tok/s (measured by llm-speed; it fits because it ran on this hardware). Quants without a run are not listed.

VRAM used: 14.0 GiB at UD-IQ3_XXS (r_e8r7fxw1ke9) peak, 8K context, of 16.0 GiB (nvidia-smi peak).

Fit sweep: what loaded and what ran out of memory

Each row is one attempt to load the model on llama.cpp at a quant and context, all layers on the GPU unless the GPU layers column or the result says otherwise. Out-of-memory and failed attempts are kept. Speeds for these configs are in the signed runs above.

Quant	Context	GPU layers	Result	Decode tok/s	Prefill tok/s	Peak VRAM GiB	Date (UTC)
UD-Q3_K_XL	8K	49 / 49	out of memory (weights)	–	–	–	2026-10-22
UD-Q3_K_XL	8K	49 / 49	ran (CPU offload; speed in the signed run)	–	–	14.3	2026-10-22
UD-IQ3_XXS	8K	49 / 49	ran (speed in the signed run)	–	–	14.0	2026-10-22
UD-IQ3_XXS	16K	49 / 49	ran (speed in the signed run)	–	–	14.2	2026-10-22

Every measured row

Run	Date (UTC)	Workload	Context	Quant	Runtime / build	Prompt tokens	Output tokens	Decode tok/s	Prefill tok/s	TTFT incl. reasoning tokens, ms	VRAM GiB	Source
r_e8r7fxw1ke9	2026-10-22	chat-short	8K	UD-IQ3_XXS	llama.cpp · commit a37cc4610	98	256	92.9	–	295	14.0 peak	Measured by llm-speed@llmspeed
r_tp8imqk28q5	2026-10-22	chat-short	16K	UD-IQ3_XXS	llama.cpp · commit a37cc4610	98	256	89.4	–	300	14.2 peak	Measured by llm-speed@llmspeed
r_acy-a8qic5j	2026-10-22	chat-shortwith CPU offload: MoE experts of 12 layers on the CPU (--n-cpu-moe 12)	8K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	98	256	15.7	–	516	14.3 peak	Measured by llm-speed@llmspeed
r_e8r7fxw1ke9	2026-10-22	chat-long	8K	UD-IQ3_XXS	llama.cpp · commit a37cc4610	3014	1024	86.7	1458.7	2066	14.0 peak	Measured by llm-speed@llmspeed
r_tp8imqk28q5	2026-10-22	chat-long	16K	UD-IQ3_XXS	llama.cpp · commit a37cc4610	3014	1024	83.3	1365.6	2207	14.2 peak	Measured by llm-speed@llmspeed
r_acy-a8qic5j	2026-10-22	chat-longwith CPU offload: MoE experts of 12 layers on the CPU (--n-cpu-moe 12)	8K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	3014	1024	10.0	840.9	3584	14.3 peak	Measured by llm-speed@llmspeed
r_e8r7fxw1ke9	2026-10-22	agent-trace	8K	UD-IQ3_XXS	llama.cpp · commit a37cc4610	3192	840	84.7	3147.3	645	14.0 peak	Measured by llm-speed@llmspeed
r_tp8imqk28q5	2026-10-22	agent-trace	16K	UD-IQ3_XXS	llama.cpp · commit a37cc4610	3192	840	82.7	3066.9	658	14.2 peak	Measured by llm-speed@llmspeed
r_acy-a8qic5j	2026-10-22	agent-tracewith CPU offload: MoE experts of 12 layers on the CPU (--n-cpu-moe 12)	8K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	3192	840	10.0	1988.6	1028	14.3 peak	Measured by llm-speed@llmspeed
r_e8r7fxw1ke9	2026-10-22	concurrent-decode	8K	UD-IQ3_XXS	llama.cpp · commit a37cc4610	1024	256	90.2	–	–	14.0 peak	Measured by llm-speed@llmspeed
r_tp8imqk28q5	2026-10-22	concurrent-decode	16K	UD-IQ3_XXS	llama.cpp · commit a37cc4610	1024	256	86.7	–	–	14.2 peak	Measured by llm-speed@llmspeed
r_acy-a8qic5j	2026-10-22	concurrent-decodewith CPU offload: MoE experts of 12 layers on the CPU (--n-cpu-moe 12)	8K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	1024	256	10.1	–	–	14.3 peak	Measured by llm-speed@llmspeed

Questions