Can an RTX 3090 run Flint 40B-A2.5B?
Only at Q3: an RTX 3090 runs Flint 40B-A2.5B at 130.8 tok/s at UD-Q3_K_XL, measured on llama.cpp with 8K context in October 2026.
3 signed runs (2 default settings, 1 CPU offload) on llm-speed. Single-request decode speed in tokens per second; the short-chat workload unless noted. All runs were measured by llm-speed (@llmspeed). Runs measured by llm-speed are credited to @llmspeed. TTFT includes reasoning tokens (time to the first token, not to the first answer); chat-short prefill is not shown because its short prompt makes it unrepresentative, so see chat-long for prefill.
Measured tok/s by quant and runtime
| Quant | Runtime | Build | Decode tok/s (best run) | Workload | Runs | Source |
|---|---|---|---|---|---|---|
| UD-Q3_K_XL | llama.cpp | commit a37cc4610 | 130.8 | chat-short | 2 | Measured by llm-speed@llmspeed |
| UD-Q4_K_XL with CPU offload | llama.cpp | commit a37cc4610 | 16.8 | chat-short | 1 | Measured by llm-speed@llmspeed |
Fastest quant that fits: UD-Q3_K_XL on llama.cpp, 130.8 tok/s (measured by llm-speed; it fits because it ran on this hardware). Quants without a run are not listed.
VRAM used: 17.8 GiB at UD-Q3_K_XL (r_-yeg4qnhjp7) peak, 8K context, of 24.0 GiB (nvidia-smi peak).
Fit sweep: what loaded and what ran out of memory
Each row is one attempt to load the model on llama.cpp at a quant and context, all layers on the GPU unless the GPU layers column or the result says otherwise. Out-of-memory and failed attempts are kept. Speeds for these configs are in the signed runs above.
| Quant | Context | GPU layers | Result | Decode tok/s | Prefill tok/s | Peak VRAM GiB | Date (UTC) |
|---|---|---|---|---|---|---|---|
| UD-Q4_K_XL | 8K | 49 / 49 | out of memory (context) | – | – | – | 2026-10-22 |
| UD-Q4_K_XL | 8K | 49 / 49 | ran (CPU offload; speed in the signed run) | – | – | 22.1 | 2026-10-22 |
| UD-Q3_K_XL | 8K | 49 / 49 | ran (speed in the signed run) | – | – | 17.8 | 2026-10-22 |
| UD-Q3_K_XL | 16K | 49 / 49 | ran (speed in the signed run) | – | – | 18.0 | 2026-10-22 |
| UD-Q3_K_XL | 32K | 49 / 49 | loads at 32K (speed not measured) | – | – | 18.4 | 2026-10-22 |
| UD-Q3_K_XL | 128K | 49 / 49 | loads at 128K (speed not measured) | – | – | 21.1 | 2026-10-22 |
Every measured row
| Run | Date (UTC) | Workload | Context | Quant | Runtime / build | Prompt tokens | Output tokens | Decode tok/s | Prefill tok/s | TTFT incl. reasoning tokens, ms | VRAM GiB | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| r_-yeg4qnhjp7 | 2026-10-22 | chat-short | 8K | UD-Q3_K_XL | llama.cpp · commit a37cc4610 | 98 | 256 | 130.8 | – | 268 | 17.8 peak | Measured by llm-speed@llmspeed |
| r_v-sdiljpm7i | 2026-10-22 | chat-short | 16K | UD-Q3_K_XL | llama.cpp · commit a37cc4610 | 98 | 256 | 128.9 | – | 264 | 18.0 peak | Measured by llm-speed@llmspeed |
| r_klxxsjjp2m- | 2026-10-22 | chat-shortwith CPU offload: MoE experts of 6 layers on the CPU (--n-cpu-moe 6) | 8K | UD-Q4_K_XL | llama.cpp · commit a37cc4610 | 98 | 256 | 16.8 | – | 489 | 22.1 peak | Measured by llm-speed@llmspeed |
| r_-yeg4qnhjp7 | 2026-10-22 | chat-long | 8K | UD-Q3_K_XL | llama.cpp · commit a37cc4610 | 3014 | 1024 | 124.8 | 2824.8 | 1067 | 17.8 peak | Measured by llm-speed@llmspeed |
| r_v-sdiljpm7i | 2026-10-22 | chat-long | 16K | UD-Q3_K_XL | llama.cpp · commit a37cc4610 | 3014 | 1024 | 123.0 | 2739.8 | 1100 | 18.0 peak | Measured by llm-speed@llmspeed |
| r_klxxsjjp2m- | 2026-10-22 | chat-longwith CPU offload: MoE experts of 6 layers on the CPU (--n-cpu-moe 6) | 8K | UD-Q4_K_XL | llama.cpp · commit a37cc4610 | 3014 | 1024 | 17.4 | 1316.2 | 2290 | 22.1 peak | Measured by llm-speed@llmspeed |
| r_-yeg4qnhjp7 | 2026-10-22 | agent-trace | 8K | UD-Q3_K_XL | llama.cpp · commit a37cc4610 | 3192 | 840 | 124.7 | 6225.9 | 331 | 17.8 peak | Measured by llm-speed@llmspeed |
| r_v-sdiljpm7i | 2026-10-22 | agent-trace | 16K | UD-Q3_K_XL | llama.cpp · commit a37cc4610 | 3192 | 840 | 124.4 | 6080.5 | 330 | 18.0 peak | Measured by llm-speed@llmspeed |
| r_klxxsjjp2m- | 2026-10-22 | agent-tracewith CPU offload: MoE experts of 6 layers on the CPU (--n-cpu-moe 6) | 8K | UD-Q4_K_XL | llama.cpp · commit a37cc4610 | 3192 | 840 | 17.0 | 2973.7 | 701 | 22.1 peak | Measured by llm-speed@llmspeed |
| r_-yeg4qnhjp7 | 2026-10-22 | concurrent-decode | 8K | UD-Q3_K_XL | llama.cpp · commit a37cc4610 | 1024 | 256 | 130.2 | – | – | 17.8 peak | Measured by llm-speed@llmspeed |
| r_v-sdiljpm7i | 2026-10-22 | concurrent-decode | 16K | UD-Q3_K_XL | llama.cpp · commit a37cc4610 | 1024 | 256 | 128.7 | – | – | 18.0 peak | Measured by llm-speed@llmspeed |
| r_klxxsjjp2m- | 2026-10-22 | concurrent-decodewith CPU offload: MoE experts of 6 layers on the CPU (--n-cpu-moe 6) | 8K | UD-Q4_K_XL | llama.cpp · commit a37cc4610 | 1024 | 256 | 16.8 | – | – | 22.1 peak | Measured by llm-speed@llmspeed |
Questions
Can an RTX 3090 run Flint 40B-A2.5B?
Only at Q3: an RTX 3090 runs Flint 40B-A2.5B at 130.8 tok/s at UD-Q3_K_XL, measured on llama.cpp with 8K context in October 2026.
How many tokens per second does Flint 40B-A2.5B get on an RTX 3090?
Decode speed measured by llm-speed, by quantization and runtime: UD-Q3_K_XL on llama.cpp: 130.8 tok/s; UD-Q4_K_XL with CPU offload on llama.cpp: 16.8 tok/s. Based on 3 runs measured by llm-speed; single-request chat-short unless noted.
What is the fastest quant of Flint 40B-A2.5B that fits on an RTX 3090?
UD-Q3_K_XL on llama.cpp was the fastest measured setting at 130.8 tok/s; it fit because it ran. Other quants are listed only where a run measured them.
How much VRAM does Flint 40B-A2.5B use on an RTX 3090?
nvidia-smi recorded a peak of 17.8 GiB of 24.0 GiB at UD-Q3_K_XL with 8K context (run r_-yeg4qnhjp7).