Can an RTX 4090 run Flint 40B-A2.5B?

Only at NVFP4 or UD-Q3_K_XL: an RTX 4090 runs Flint 40B-A2.5B at 179.4 tok/s at NVFP4 (tested at 8K only), measured on llama.cpp in October 2026.

4 signed runs (3 default settings, 1 CPU offload) on llm-speed. Single-request decode speed in tokens per second; the short-chat workload unless noted. All runs were measured by llm-speed (@llmspeed). Runs measured by llm-speed are credited to @llmspeed. TTFT includes reasoning tokens (time to the first token, not to the first answer); chat-short prefill is not shown because its short prompt makes it unrepresentative, so see chat-long for prefill.

Measured tok/s by quant and runtime

Quant	Runtime	Build	Decode tok/s (best run)	Workload	Runs	Source
NVFP4	llama.cpp	commit a37cc4610	179.4	chat-short	1	Measured by llm-speed@llmspeed
UD-Q3_K_XL	llama.cpp	commit a37cc4610	169.2	chat-short	2	Measured by llm-speed@llmspeed
UD-Q4_K_XL with CPU offload	llama.cpp	commit a37cc4610	39.8	chat-short	1	Measured by llm-speed@llmspeed

Fastest quant that fits: NVFP4 on llama.cpp, 179.4 tok/s (measured by llm-speed; it fits because it ran on this hardware). Quants without a run are not listed.

Why NVFP4 beats Q3 here: it measured 179.4 tok/s vs 168.0 for UD-Q3_K_XL (8K context, same llama.cpp build). The RTX 4090 (Ada) has no native FP4 tensor cores, so this is not an FP4 hardware speedup; it reflects how llama.cpp's kernels handle each format on this card.

VRAM used: 21.7 GiB at NVFP4 (r_0i0a9uq6fdx), 17.9 GiB at UD-Q3_K_XL (r_r7gzl4lmmdc) peak, 8K context, of 24.0 GiB (nvidia-smi peak).

Fit sweep: what loaded and what ran out of memory

Each row is one attempt to load the model on llama.cpp at a quant and context, all layers on the GPU unless the GPU layers column or the result says otherwise. Out-of-memory and failed attempts are kept. Speeds for these configs are in the signed runs above.

Quant	Context	GPU layers	Result	Decode tok/s	Prefill tok/s	Peak VRAM GiB	Date (UTC)
UD-Q4_K_XL	8K	49 / 49	out of memory (context)	–	–	–	2026-10-22
UD-Q4_K_XL	8K	49 / 49	ran (CPU offload; speed in the signed run)	–	–	22.2	2026-10-22
UD-Q4_K_XL	32K	49 / 49	out of memory (context)	–	–	–	2026-10-22
NVFP4	8K	49 / 49	ran (speed in the signed run)	–	–	21.7	2026-10-22
UD-Q3_K_XL	8K	49 / 49	ran (speed in the signed run)	–	–	17.9	2026-10-22
UD-Q3_K_XL	16K	49 / 49	ran (speed in the signed run)	–	–	18.1	2026-10-22
UD-Q3_K_XL	32K	49 / 49	loads at 32K (speed not measured)	–	–	18.5	2026-10-22
UD-Q3_K_XL	128K	49 / 49	loads at 128K (speed not measured)	–	–	21.3	2026-10-22
UD-Q3_K_XL	256K	49 / 49	out of memory (context)	–	–	–	2026-10-22

Every measured row

Run	Date (UTC)	Workload	Context	Quant	Runtime / build	Prompt tokens	Output tokens	Decode tok/s	Prefill tok/s	TTFT incl. reasoning tokens, ms	VRAM GiB	Source
r_0i0a9uq6fdx	2026-10-22	chat-short	8K	NVFP4	llama.cpp · commit a37cc4610	98	256	179.4	–	232	21.7 peak	Measured by llm-speed@llmspeed
r_sw92j6kg7js	2026-10-22	chat-short	16K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	98	256	169.2	–	222	18.1 peak	Measured by llm-speed@llmspeed
r_r7gzl4lmmdc	2026-10-22	chat-short	8K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	98	256	168.0	–	201	17.9 peak	Measured by llm-speed@llmspeed
r_wap61ewkpzm	2026-10-22	chat-shortwith CPU offload: MoE experts of 6 layers on the CPU (--n-cpu-moe 6)	8K	UD-Q4_K_XL	llama.cpp · commit a37cc4610	98	256	39.8	–	356	22.2 peak	Measured by llm-speed@llmspeed
r_0i0a9uq6fdx	2026-10-22	chat-long	8K	NVFP4	llama.cpp · commit a37cc4610	3014	1024	167.2	4992.3	604	21.7 peak	Measured by llm-speed@llmspeed
r_sw92j6kg7js	2026-10-22	chat-long	16K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	3014	1024	157.8	5779.1	522	18.1 peak	Measured by llm-speed@llmspeed
r_r7gzl4lmmdc	2026-10-22	chat-long	8K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	3014	1024	157.6	5789.2	521	17.9 peak	Measured by llm-speed@llmspeed
r_wap61ewkpzm	2026-10-22	chat-longwith CPU offload: MoE experts of 6 layers on the CPU (--n-cpu-moe 6)	8K	UD-Q4_K_XL	llama.cpp · commit a37cc4610	3014	1024	40.7	2006.3	1502	22.2 peak	Measured by llm-speed@llmspeed
r_0i0a9uq6fdx	2026-10-22	agent-trace	8K	NVFP4	llama.cpp · commit a37cc4610	3192	840	168.5	8953.3	214	21.7 peak	Measured by llm-speed@llmspeed
r_sw92j6kg7js	2026-10-22	agent-trace	16K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	3192	840	160.4	10368.2	184	18.1 peak	Measured by llm-speed@llmspeed
r_r7gzl4lmmdc	2026-10-22	agent-trace	8K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	3192	840	160.3	10408.2	184	17.9 peak	Measured by llm-speed@llmspeed
r_wap61ewkpzm	2026-10-22	agent-tracewith CPU offload: MoE experts of 6 layers on the CPU (--n-cpu-moe 6)	8K	UD-Q4_K_XL	llama.cpp · commit a37cc4610	3192	840	41.4	4498.6	494	22.2 peak	Measured by llm-speed@llmspeed
r_0i0a9uq6fdx	2026-10-22	concurrent-decode	8K	NVFP4	llama.cpp · commit a37cc4610	1024	256	176.1	–	–	21.7 peak	Measured by llm-speed@llmspeed
r_r7gzl4lmmdc	2026-10-22	concurrent-decode	8K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	1024	256	166.5	–	–	17.9 peak	Measured by llm-speed@llmspeed
r_sw92j6kg7js	2026-10-22	concurrent-decode	16K	UD-Q3_K_XL	llama.cpp · commit a37cc4610	1024	256	166.2	–	–	18.1 peak	Measured by llm-speed@llmspeed
r_wap61ewkpzm	2026-10-22	concurrent-decodewith CPU offload: MoE experts of 6 layers on the CPU (--n-cpu-moe 6)	8K	UD-Q4_K_XL	llama.cpp · commit a37cc4610	1024	256	42.7	–	–	22.2 peak	Measured by llm-speed@llmspeed

Questions