Can an RTX A4000 run Muse-Glimmer-30B?
The answer
Yes: an RTX A4000 runs Muse-Glimmer-30B at 23.3 tok/s at UD-IQ3_XXS, measured on llama.cpp with 8K context in October 2026; UD-IQ3_XXS loaded fully on the GPU at up to 128K context.
14 signed runs (7 f16 KV, 7 q8_0 KV) on llm-speed. Single-request decode speed in tokens per second; the short-chat workload unless noted. All runs were measured by llm-speed (@llmspeed). Runs measured by llm-speed are credited to @llmspeed. TTFT includes reasoning tokens (time to the first token, not to the first answer); chat-short prefill is not shown because its short prompt makes it unrepresentative, so see chat-long for prefill.
Measured tok/s by quant and runtime
| Quant | Runtime | Build | Decode tok/s (best run) | Workload | Runs | Source |
|---|---|---|---|---|---|---|
| UD-IQ3_XXS | llama.cpp | commit a37cc46 | 23.3 | chat-short | 7 | Measured by llm-speed@llmspeed |
| UD-IQ3_XXSq8_0 KV cache (-ctk q8_0 -ctv q8_0) (counted separately) | llama.cpp | commit a37cc46 | 21.8 | chat-short | 7 | Measured by llm-speed@llmspeed |
Fastest quant that fits: UD-IQ3_XXS on llama.cpp, 23.3 tok/s (measured by llm-speed; it fits because it ran on this hardware). Quants without a run are not listed.
Context: unsloth's UD-IQ3_XXS GGUF (13.1 GB) loaded fully on the GPU at every context from 8K to the model's 128K maximum, with either KV cache type: 13,977 MiB peak at 128K with the default f16 KV cache and 13,067 MiB with a q8_0 KV cache, of the card's 16,376 MiB. Speeds are short prompts at each configured context, not a filled 128K window. q8_0 KV cache: those runs passed -ctk q8_0 -ctv q8_0 to llama-server, and the CLI records that setting as declared rather than reading it back from the server; their 192 to 910 MiB lower peaks are the independent sign it took effect. The fit table lists the f16 loads. No control model was measured on this card, so compare these speeds only with each other. Text only: the vision projector was not loaded.
VRAM used: 12.0 GiB at UD-IQ3_XXS (r_u_kr0v-wwde) peak, 8K context, of 16.0 GiB (nvidia-smi peak).
Check whether a model fits your GPU with the VRAM calculator.
Fit sweep: what loaded and what ran out of memory
Each row is one attempt to load the model on llama.cpp at a quant and context, all layers on the GPU unless the GPU layers column or the result says otherwise. Out-of-memory and failed attempts are kept. Speeds for these configs are in the signed runs above.
| Quant | Context | GPU layers | Result | Decode tok/s | Prefill tok/s | Peak VRAM GiB | Date (UTC) |
|---|---|---|---|---|---|---|---|
| UD-IQ3_XXS | 8K | all | ran (speed in the signed run) | – | – | 12.0 | 2026-10-22 |
| UD-IQ3_XXS | 16K | all | ran (speed in the signed run) | – | – | 12.3 | 2026-10-22 |
| UD-IQ3_XXS | 32K | all | ran (speed in the signed run) | – | – | 12.3 | 2026-10-22 |
| UD-IQ3_XXS | 48K | all | ran (speed in the signed run) | – | – | 12.6 | 2026-10-22 |
| UD-IQ3_XXS | 64K | all | ran (speed in the signed run) | – | – | 12.9 | 2026-10-22 |
| UD-IQ3_XXS | 96K | all | ran (speed in the signed run) | – | – | 13.2 | 2026-10-22 |
| UD-IQ3_XXS | 128K | all | ran (speed in the signed run) | – | – | 13.6 | 2026-10-22 |
Every measured row
| Run | Date (UTC) | Workload | Context | Quant | Runtime / build | Prompt tokens | Output tokens | Decode tok/s | Prefill tok/s | TTFT incl. reasoning tokens, ms | VRAM GiB | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| r_u_kr0v-wwde | 2026-10-22 | chat-short | 8K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 23.3 | – | 581 | 12.0 peak | Measured by llm-speed@llmspeed |
| r_1lyb_mqmdik | 2026-10-22 | chat-short | 128K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.9 | – | 604 | 13.6 peak | Measured by llm-speed@llmspeed |
| r_0o1xfrrn-ch | 2026-10-22 | chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 96K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.8 | – | 583 | 12.5 peak | Measured by llm-speed@llmspeed |
| r_x8mqtlf166j | 2026-10-22 | chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 48K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.7 | – | 563 | 12.1 peak | Measured by llm-speed@llmspeed |
| r_dldd3d54262 | 2026-10-22 | chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 16K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.7 | – | 608 | 12.0 peak | Measured by llm-speed@llmspeed |
| r_b8y4oh7e0-z | 2026-10-22 | chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 8K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.7 | – | 567 | 11.8 peak | Measured by llm-speed@llmspeed |
| r_crqe50ubylk | 2026-10-22 | chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 128K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.7 | – | 626 | 12.8 peak | Measured by llm-speed@llmspeed |
| r_zoeatakzums | 2026-10-22 | chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 32K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.6 | – | 566 | 12.0 peak | Measured by llm-speed@llmspeed |
| r_pf4ye8_4xmc | 2026-10-22 | chat-short | 96K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.6 | – | 580 | 13.2 peak | Measured by llm-speed@llmspeed |
| r_aufjud7duqr | 2026-10-22 | chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 64K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.5 | – | 578 | 12.4 peak | Measured by llm-speed@llmspeed |
| r_j9yaui8ovvy | 2026-10-22 | chat-short | 64K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.5 | – | 561 | 12.9 peak | Measured by llm-speed@llmspeed |
| r_ntsos2cxwja | 2026-10-22 | chat-short | 32K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.5 | – | 571 | 12.3 peak | Measured by llm-speed@llmspeed |
| r_y8qy4yd5p7i | 2026-10-22 | chat-short | 48K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.5 | – | 567 | 12.6 peak | Measured by llm-speed@llmspeed |
| r_y9y09s77lfa | 2026-10-22 | chat-short | 16K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 154 | 256 | 21.4 | – | 600 | 12.3 peak | Measured by llm-speed@llmspeed |
| r_u_kr0v-wwde | 2026-10-22 | chat-long | 8K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 22.3 | 722.0 | 4348 | 12.0 peak | Measured by llm-speed@llmspeed |
| r_1lyb_mqmdik | 2026-10-22 | chat-long | 128K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 21.2 | 698.3 | 4495 | 13.6 peak | Measured by llm-speed@llmspeed |
| r_pf4ye8_4xmc | 2026-10-22 | chat-long | 96K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.9 | 688.0 | 4562 | 13.2 peak | Measured by llm-speed@llmspeed |
| r_y8qy4yd5p7i | 2026-10-22 | chat-long | 48K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.8 | 691.1 | 4542 | 12.6 peak | Measured by llm-speed@llmspeed |
| r_y9y09s77lfa | 2026-10-22 | chat-long | 16K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.8 | 687.0 | 4569 | 12.3 peak | Measured by llm-speed@llmspeed |
| r_j9yaui8ovvy | 2026-10-22 | chat-long | 64K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.8 | 681.3 | 4608 | 12.9 peak | Measured by llm-speed@llmspeed |
| r_ntsos2cxwja | 2026-10-22 | chat-long | 32K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.8 | 684.1 | 4588 | 12.3 peak | Measured by llm-speed@llmspeed |
| r_b8y4oh7e0-z | 2026-10-22 | chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 8K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.1 | 695.4 | 4514 | 11.8 peak | Measured by llm-speed@llmspeed |
| r_0o1xfrrn-ch | 2026-10-22 | chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 96K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.1 | 695.0 | 4517 | 12.5 peak | Measured by llm-speed@llmspeed |
| r_x8mqtlf166j | 2026-10-22 | chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 48K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.1 | 691.7 | 4538 | 12.1 peak | Measured by llm-speed@llmspeed |
| r_dldd3d54262 | 2026-10-22 | chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 16K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.1 | 697.9 | 4497 | 12.0 peak | Measured by llm-speed@llmspeed |
| r_zoeatakzums | 2026-10-22 | chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 32K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.0 | 697.4 | 4501 | 12.0 peak | Measured by llm-speed@llmspeed |
| r_aufjud7duqr | 2026-10-22 | chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 64K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.0 | 690.2 | 4548 | 12.4 peak | Measured by llm-speed@llmspeed |
| r_crqe50ubylk | 2026-10-22 | chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 128K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3139 | 1024 | 20.0 | 700.7 | 4480 | 12.8 peak | Measured by llm-speed@llmspeed |
| r_pf4ye8_4xmc | 2026-10-22 | agent-trace | 96K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 20.7 | 1638.2 | 1201 | 13.2 peak | Measured by llm-speed@llmspeed |
| r_1lyb_mqmdik | 2026-10-22 | agent-trace | 128K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 20.3 | 1665.1 | 1255 | 13.6 peak | Measured by llm-speed@llmspeed |
| r_b8y4oh7e0-z | 2026-10-22 | agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 8K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 20.2 | 1658.2 | 1003 | 11.8 peak | Measured by llm-speed@llmspeed |
| r_0o1xfrrn-ch | 2026-10-22 | agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 96K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 20.2 | 1626.9 | 1013 | 12.5 peak | Measured by llm-speed@llmspeed |
| r_u_kr0v-wwde | 2026-10-22 | agent-trace | 8K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 20.2 | 1636.4 | 1012 | 12.0 peak | Measured by llm-speed@llmspeed |
| r_y9y09s77lfa | 2026-10-22 | agent-trace | 16K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 20.1 | 1655.8 | 1014 | 12.3 peak | Measured by llm-speed@llmspeed |
| r_dldd3d54262 | 2026-10-22 | agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 16K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 20.0 | 1659.6 | 944 | 12.0 peak | Measured by llm-speed@llmspeed |
| r_y8qy4yd5p7i | 2026-10-22 | agent-trace | 48K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 19.7 | 1639.3 | 997 | 12.6 peak | Measured by llm-speed@llmspeed |
| r_j9yaui8ovvy | 2026-10-22 | agent-trace | 64K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 19.5 | 1655.9 | 1009 | 12.9 peak | Measured by llm-speed@llmspeed |
| r_ntsos2cxwja | 2026-10-22 | agent-trace | 32K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 19.4 | 1628.3 | 1273 | 12.3 peak | Measured by llm-speed@llmspeed |
| r_x8mqtlf166j | 2026-10-22 | agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 48K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 19.4 | 1668.3 | 1138 | 12.1 peak | Measured by llm-speed@llmspeed |
| r_crqe50ubylk | 2026-10-22 | agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 128K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 19.3 | 1695.8 | 980 | 12.8 peak | Measured by llm-speed@llmspeed |
| r_aufjud7duqr | 2026-10-22 | agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 64K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 19.0 | 1676.7 | 1190 | 12.4 peak | Measured by llm-speed@llmspeed |
| r_zoeatakzums | 2026-10-22 | agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 32K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 3262 | 840 | 18.5 | 1667.6 | 968 | 12.0 peak | Measured by llm-speed@llmspeed |
| r_u_kr0v-wwde | 2026-10-22 | concurrent-decode | 8K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 21.6 | – | – | 12.0 peak | Measured by llm-speed@llmspeed |
| r_1lyb_mqmdik | 2026-10-22 | concurrent-decode | 128K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 21.1 | – | – | 13.6 peak | Measured by llm-speed@llmspeed |
| r_pf4ye8_4xmc | 2026-10-22 | concurrent-decode | 96K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 21.0 | – | – | 13.2 peak | Measured by llm-speed@llmspeed |
| r_0o1xfrrn-ch | 2026-10-22 | concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 96K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 20.9 | – | – | 12.5 peak | Measured by llm-speed@llmspeed |
| r_y9y09s77lfa | 2026-10-22 | concurrent-decode | 16K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 20.8 | – | – | 12.3 peak | Measured by llm-speed@llmspeed |
| r_x8mqtlf166j | 2026-10-22 | concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 48K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 20.8 | – | – | 12.1 peak | Measured by llm-speed@llmspeed |
| r_aufjud7duqr | 2026-10-22 | concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 64K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 20.7 | – | – | 12.4 peak | Measured by llm-speed@llmspeed |
| r_dldd3d54262 | 2026-10-22 | concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 16K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 20.7 | – | – | 12.0 peak | Measured by llm-speed@llmspeed |
| r_b8y4oh7e0-z | 2026-10-22 | concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 8K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 20.7 | – | – | 11.8 peak | Measured by llm-speed@llmspeed |
| r_j9yaui8ovvy | 2026-10-22 | concurrent-decode | 64K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 20.6 | – | – | 12.9 peak | Measured by llm-speed@llmspeed |
| r_crqe50ubylk | 2026-10-22 | concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 128K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 20.6 | – | – | 12.8 peak | Measured by llm-speed@llmspeed |
| r_ntsos2cxwja | 2026-10-22 | concurrent-decode | 32K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 20.6 | – | – | 12.3 peak | Measured by llm-speed@llmspeed |
| r_zoeatakzums | 2026-10-22 | concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0) | 32K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 20.6 | – | – | 12.0 peak | Measured by llm-speed@llmspeed |
| r_y8qy4yd5p7i | 2026-10-22 | concurrent-decode | 48K | UD-IQ3_XXS | llama.cpp0.7.0-dev (build 11803, commit a37cc4610) | 1024 | 256 | 20.6 | – | – | 12.6 peak | Measured by llm-speed@llmspeed |
Questions
Can an RTX A4000 run Muse-Glimmer-30B?
Yes: an RTX A4000 runs Muse-Glimmer-30B at 23.3 tok/s at UD-IQ3_XXS, measured on llama.cpp with 8K context in October 2026; UD-IQ3_XXS loaded fully on the GPU at up to 128K context.
How many tokens per second does Muse-Glimmer-30B get on an RTX A4000?
Decode speed measured by llm-speed, by quantization and runtime: UD-IQ3_XXS on llama.cpp: 23.3 tok/s; UD-IQ3_XXS (q8_0 KV cache (-ctk q8_0 -ctv q8_0)) on llama.cpp: 21.8 tok/s. Based on 7 runs measured by llm-speed plus 7 runs with a non-default configuration, listed separately; single-request chat-short unless noted.
What is the fastest quant of Muse-Glimmer-30B that fits on an RTX A4000?
UD-IQ3_XXS on llama.cpp was the fastest measured setting at 23.3 tok/s; it fit because it ran. Other quants are listed only where a run measured them.
How much VRAM does Muse-Glimmer-30B use on an RTX A4000?
nvidia-smi recorded a peak of 12.0 GiB of 16.0 GiB at UD-IQ3_XXS with 8K context (run r_u_kr0v-wwde).
Sources and related pages
- Source runs: r_u_kr0v-wwde, r_1lyb_mqmdik, r_0o1xfrrn-ch, r_x8mqtlf166j, r_dldd3d54262, r_b8y4oh7e0-z, r_crqe50ubylk, r_zoeatakzums, r_pf4ye8_4xmc, r_aufjud7duqr, r_j9yaui8ovvy, r_ntsos2cxwja, r_y8qy4yd5p7i, r_y9y09s77lfa
- Muse-Glimmer-30B: every measured GPU and Mac
- RTX A4000: every measured model
- How runs are measured · open dataset (CC BY 4.0)