Skip to content
llm-speed

Can an RTX A4000 run Muse-Glimmer-30B?

The answer

Yes: an RTX A4000 runs Muse-Glimmer-30B at 23.3 tok/s at UD-IQ3_XXS, measured on llama.cpp with 8K context in October 2026; UD-IQ3_XXS loaded fully on the GPU at up to 128K context.

14 signed runs (7 f16 KV, 7 q8_0 KV) on llm-speed. Single-request decode speed in tokens per second; the short-chat workload unless noted. All runs were measured by llm-speed (@llmspeed). Runs measured by llm-speed are credited to @llmspeed. TTFT includes reasoning tokens (time to the first token, not to the first answer); chat-short prefill is not shown because its short prompt makes it unrepresentative, so see chat-long for prefill.

Measured tok/s by quant and runtime

QuantRuntimeBuildDecode tok/s (best run)WorkloadRunsSource
UD-IQ3_XXSllama.cppcommit a37cc4623.3chat-short7Measured by llm-speed@llmspeed
UD-IQ3_XXSq8_0 KV cache (-ctk q8_0 -ctv q8_0) (counted separately)llama.cppcommit a37cc4621.8chat-short7Measured by llm-speed@llmspeed

Fastest quant that fits: UD-IQ3_XXS on llama.cpp, 23.3 tok/s (measured by llm-speed; it fits because it ran on this hardware). Quants without a run are not listed.

Context: unsloth's UD-IQ3_XXS GGUF (13.1 GB) loaded fully on the GPU at every context from 8K to the model's 128K maximum, with either KV cache type: 13,977 MiB peak at 128K with the default f16 KV cache and 13,067 MiB with a q8_0 KV cache, of the card's 16,376 MiB. Speeds are short prompts at each configured context, not a filled 128K window. q8_0 KV cache: those runs passed -ctk q8_0 -ctv q8_0 to llama-server, and the CLI records that setting as declared rather than reading it back from the server; their 192 to 910 MiB lower peaks are the independent sign it took effect. The fit table lists the f16 loads. No control model was measured on this card, so compare these speeds only with each other. Text only: the vision projector was not loaded.

VRAM used: 12.0 GiB at UD-IQ3_XXS (r_u_kr0v-wwde) peak, 8K context, of 16.0 GiB (nvidia-smi peak).

Check whether a model fits your GPU with the VRAM calculator.

Fit sweep: what loaded and what ran out of memory

Each row is one attempt to load the model on llama.cpp at a quant and context, all layers on the GPU unless the GPU layers column or the result says otherwise. Out-of-memory and failed attempts are kept. Speeds for these configs are in the signed runs above.

QuantContextGPU layersResultDecode tok/sPrefill tok/sPeak VRAM GiBDate (UTC)
UD-IQ3_XXS8Kallran (speed in the signed run)––12.02026-10-22
UD-IQ3_XXS16Kallran (speed in the signed run)––12.32026-10-22
UD-IQ3_XXS32Kallran (speed in the signed run)––12.32026-10-22
UD-IQ3_XXS48Kallran (speed in the signed run)––12.62026-10-22
UD-IQ3_XXS64Kallran (speed in the signed run)––12.92026-10-22
UD-IQ3_XXS96Kallran (speed in the signed run)––13.22026-10-22
UD-IQ3_XXS128Kallran (speed in the signed run)––13.62026-10-22

Every measured row

RunDate (UTC)WorkloadContextQuantRuntime / buildPrompt tokensOutput tokensDecode tok/sPrefill tok/sTTFT incl. reasoning tokens, msVRAM GiBSource
r_u_kr0v-wwde2026-10-22chat-short8KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425623.3–58112.0 peakMeasured by llm-speed@llmspeed
r_1lyb_mqmdik2026-10-22chat-short128KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.9–60413.6 peakMeasured by llm-speed@llmspeed
r_0o1xfrrn-ch2026-10-22chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0)96KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.8–58312.5 peakMeasured by llm-speed@llmspeed
r_x8mqtlf166j2026-10-22chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0)48KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.7–56312.1 peakMeasured by llm-speed@llmspeed
r_dldd3d542622026-10-22chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0)16KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.7–60812.0 peakMeasured by llm-speed@llmspeed
r_b8y4oh7e0-z2026-10-22chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0)8KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.7–56711.8 peakMeasured by llm-speed@llmspeed
r_crqe50ubylk2026-10-22chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0)128KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.7–62612.8 peakMeasured by llm-speed@llmspeed
r_zoeatakzums2026-10-22chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0)32KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.6–56612.0 peakMeasured by llm-speed@llmspeed
r_pf4ye8_4xmc2026-10-22chat-short96KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.6–58013.2 peakMeasured by llm-speed@llmspeed
r_aufjud7duqr2026-10-22chat-shortq8_0 KV cache (-ctk q8_0 -ctv q8_0)64KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.5–57812.4 peakMeasured by llm-speed@llmspeed
r_j9yaui8ovvy2026-10-22chat-short64KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.5–56112.9 peakMeasured by llm-speed@llmspeed
r_ntsos2cxwja2026-10-22chat-short32KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.5–57112.3 peakMeasured by llm-speed@llmspeed
r_y8qy4yd5p7i2026-10-22chat-short48KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.5–56712.6 peakMeasured by llm-speed@llmspeed
r_y9y09s77lfa2026-10-22chat-short16KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)15425621.4–60012.3 peakMeasured by llm-speed@llmspeed
r_u_kr0v-wwde2026-10-22chat-long8KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102422.3722.0434812.0 peakMeasured by llm-speed@llmspeed
r_1lyb_mqmdik2026-10-22chat-long128KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102421.2698.3449513.6 peakMeasured by llm-speed@llmspeed
r_pf4ye8_4xmc2026-10-22chat-long96KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.9688.0456213.2 peakMeasured by llm-speed@llmspeed
r_y8qy4yd5p7i2026-10-22chat-long48KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.8691.1454212.6 peakMeasured by llm-speed@llmspeed
r_y9y09s77lfa2026-10-22chat-long16KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.8687.0456912.3 peakMeasured by llm-speed@llmspeed
r_j9yaui8ovvy2026-10-22chat-long64KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.8681.3460812.9 peakMeasured by llm-speed@llmspeed
r_ntsos2cxwja2026-10-22chat-long32KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.8684.1458812.3 peakMeasured by llm-speed@llmspeed
r_b8y4oh7e0-z2026-10-22chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0)8KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.1695.4451411.8 peakMeasured by llm-speed@llmspeed
r_0o1xfrrn-ch2026-10-22chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0)96KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.1695.0451712.5 peakMeasured by llm-speed@llmspeed
r_x8mqtlf166j2026-10-22chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0)48KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.1691.7453812.1 peakMeasured by llm-speed@llmspeed
r_dldd3d542622026-10-22chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0)16KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.1697.9449712.0 peakMeasured by llm-speed@llmspeed
r_zoeatakzums2026-10-22chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0)32KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.0697.4450112.0 peakMeasured by llm-speed@llmspeed
r_aufjud7duqr2026-10-22chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0)64KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.0690.2454812.4 peakMeasured by llm-speed@llmspeed
r_crqe50ubylk2026-10-22chat-longq8_0 KV cache (-ctk q8_0 -ctv q8_0)128KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)3139102420.0700.7448012.8 peakMeasured by llm-speed@llmspeed
r_pf4ye8_4xmc2026-10-22agent-trace96KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284020.71638.2120113.2 peakMeasured by llm-speed@llmspeed
r_1lyb_mqmdik2026-10-22agent-trace128KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284020.31665.1125513.6 peakMeasured by llm-speed@llmspeed
r_b8y4oh7e0-z2026-10-22agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0)8KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284020.21658.2100311.8 peakMeasured by llm-speed@llmspeed
r_0o1xfrrn-ch2026-10-22agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0)96KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284020.21626.9101312.5 peakMeasured by llm-speed@llmspeed
r_u_kr0v-wwde2026-10-22agent-trace8KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284020.21636.4101212.0 peakMeasured by llm-speed@llmspeed
r_y9y09s77lfa2026-10-22agent-trace16KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284020.11655.8101412.3 peakMeasured by llm-speed@llmspeed
r_dldd3d542622026-10-22agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0)16KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284020.01659.694412.0 peakMeasured by llm-speed@llmspeed
r_y8qy4yd5p7i2026-10-22agent-trace48KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284019.71639.399712.6 peakMeasured by llm-speed@llmspeed
r_j9yaui8ovvy2026-10-22agent-trace64KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284019.51655.9100912.9 peakMeasured by llm-speed@llmspeed
r_ntsos2cxwja2026-10-22agent-trace32KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284019.41628.3127312.3 peakMeasured by llm-speed@llmspeed
r_x8mqtlf166j2026-10-22agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0)48KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284019.41668.3113812.1 peakMeasured by llm-speed@llmspeed
r_crqe50ubylk2026-10-22agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0)128KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284019.31695.898012.8 peakMeasured by llm-speed@llmspeed
r_aufjud7duqr2026-10-22agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0)64KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284019.01676.7119012.4 peakMeasured by llm-speed@llmspeed
r_zoeatakzums2026-10-22agent-traceq8_0 KV cache (-ctk q8_0 -ctv q8_0)32KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)326284018.51667.696812.0 peakMeasured by llm-speed@llmspeed
r_u_kr0v-wwde2026-10-22concurrent-decode8KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425621.6––12.0 peakMeasured by llm-speed@llmspeed
r_1lyb_mqmdik2026-10-22concurrent-decode128KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425621.1––13.6 peakMeasured by llm-speed@llmspeed
r_pf4ye8_4xmc2026-10-22concurrent-decode96KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425621.0––13.2 peakMeasured by llm-speed@llmspeed
r_0o1xfrrn-ch2026-10-22concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0)96KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425620.9––12.5 peakMeasured by llm-speed@llmspeed
r_y9y09s77lfa2026-10-22concurrent-decode16KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425620.8––12.3 peakMeasured by llm-speed@llmspeed
r_x8mqtlf166j2026-10-22concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0)48KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425620.8––12.1 peakMeasured by llm-speed@llmspeed
r_aufjud7duqr2026-10-22concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0)64KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425620.7––12.4 peakMeasured by llm-speed@llmspeed
r_dldd3d542622026-10-22concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0)16KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425620.7––12.0 peakMeasured by llm-speed@llmspeed
r_b8y4oh7e0-z2026-10-22concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0)8KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425620.7––11.8 peakMeasured by llm-speed@llmspeed
r_j9yaui8ovvy2026-10-22concurrent-decode64KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425620.6––12.9 peakMeasured by llm-speed@llmspeed
r_crqe50ubylk2026-10-22concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0)128KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425620.6––12.8 peakMeasured by llm-speed@llmspeed
r_ntsos2cxwja2026-10-22concurrent-decode32KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425620.6––12.3 peakMeasured by llm-speed@llmspeed
r_zoeatakzums2026-10-22concurrent-decodeq8_0 KV cache (-ctk q8_0 -ctv q8_0)32KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425620.6––12.0 peakMeasured by llm-speed@llmspeed
r_y8qy4yd5p7i2026-10-22concurrent-decode48KUD-IQ3_XXSllama.cpp0.7.0-dev (build 11803, commit a37cc4610)102425620.6––12.6 peakMeasured by llm-speed@llmspeed

Questions

Can an RTX A4000 run Muse-Glimmer-30B?

Yes: an RTX A4000 runs Muse-Glimmer-30B at 23.3 tok/s at UD-IQ3_XXS, measured on llama.cpp with 8K context in October 2026; UD-IQ3_XXS loaded fully on the GPU at up to 128K context.

How many tokens per second does Muse-Glimmer-30B get on an RTX A4000?

Decode speed measured by llm-speed, by quantization and runtime: UD-IQ3_XXS on llama.cpp: 23.3 tok/s; UD-IQ3_XXS (q8_0 KV cache (-ctk q8_0 -ctv q8_0)) on llama.cpp: 21.8 tok/s. Based on 7 runs measured by llm-speed plus 7 runs with a non-default configuration, listed separately; single-request chat-short unless noted.

What is the fastest quant of Muse-Glimmer-30B that fits on an RTX A4000?

UD-IQ3_XXS on llama.cpp was the fastest measured setting at 23.3 tok/s; it fit because it ran. Other quants are listed only where a run measured them.

How much VRAM does Muse-Glimmer-30B use on an RTX A4000?

nvidia-smi recorded a peak of 12.0 GiB of 16.0 GiB at UD-IQ3_XXS with 8K context (run r_u_kr0v-wwde).

Sources and related pages