Qwen3.8-27B
6 workload results across 1 hardware configuration.
Original test · 5 September 2026 · three repetitions
How fast is Qwen3.8-27B on an RTX 5090?
We measured 65.7 tokens per second in short chat with Q4_K_M quantization, using one RTX 5090 with 32 GB VRAM and llama.cpp. Median time to the first token was 117.5 ms with the model already loaded. These are single-user, thinking-off results.
| Workload | Decode tok/s | First token (ms) | Prompt tokens | Output tokens |
|---|---|---|---|---|
| chat-short | 65.7 (65.4–65.7) | 117.5 (117.1–131.7) | 114 | 256 |
| chat-long | 65.1 (65.1–65.1) | 909.4 (908.0–910.5) | 3,184 | 874 |
Does it fit in 32 GB of VRAM?
Yes, this tested configuration offloaded all 65/65 layers to the GPU. The runtime reported 18.62 GiB (19,071 MiB) of GPU allocation for model, context and compute buffers combined. The model file is 18.97 GB on disk.
Allocation is from the runtime log, not a measurement of peak whole-device memory. A 16,384-token context was configured; the prompts above do not test a full 16K context. Other idle services remained GPU-resident.
What do these results tell you?
This is a baseline for local text generation with one active request. It helps estimate response speed at the tested prompt lengths. Prompt caching and thinking were disabled; the model stayed loaded between runs.
It does not establish answer quality, coding accuracy, vision performance, cold-start time or multi-user throughput. Different quantization, context lengths and runtime optimizations need separate measurements.
Tested settings and reproducibility
Hardware: RTX 5090 (32 GB), AMD Ryzen 7 9850X3D, 30 GB system RAM reported by WSL. Runtime: llama.cpp commit 9725a31 with CUDA.
Artifact: Qwen3.8-27B-Q4_K_M.gguf, pinned revision 0669b98607d47046c7c2b3f801011d54a08cfccf. SHA-256: 31629f53165ab6a7dad8c9847dcfd1fdf55829dac1e6e748f4a68581b0033d34.
Server configuration (use a free local port and the path to your verified model file):
llama-server --model Qwen3.8-27B-Q4_K_M.gguf --alias Qwen3.8-27B-Q4_K_M.gguf \
--host 127.0.0.1 --port 8082 --n-gpu-layers 999 \
--ctx-size 16384 --parallel 1 --no-cache-prompt --cache-ram 0 \
--reasoning off --temp 0 --seed 42Measurements used suite-v1 chat-short and chat-long, with output caps of 256 and 1,024 tokens. The table reports actual token counts. Decode speed and first-token latency use the benchmark client's stream timing.
The client was an isolated copy of llm-speed 0.0.5 with fixes for underscore-containing quantization names and streamed token usage. Prompts and timing formulas were unchanged. Installing the published CLI alone may not reproduce that exact client. The configuration above documents the server; it is not a complete benchmark command.
Five earlier submissions affected by a competing service restart were excluded. Only the three accepted repetitions linked below contribute to this summary.
Inspect the measured runs
Also tested on this GPU: Gemma 4 12B IT QAT. Tokenizers and output lengths differ, so speed alone does not rank the models' usefulness.
Local runs (6 runs)
Runs from contributors' own machines via MLX, llama.cpp, vLLM, exllamav2, or ollama. Signed on the submitter's hardware.
RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp@1 (9725a31) | Q4_K_M | 65.66tok/s | 969.8tok/s | 118ms | r_dz5a3lzxhrj |
| chat-long | llama.cpp@1 (9725a31) | Q4_K_M | 65.10tok/s | 3,496.8tok/s | 911ms | r_dz5a3lzxhrj |
| chat-short | llama.cpp@1 (9725a31) | Q4_K_M | 65.73tok/s | 973.6tok/s | 117ms | r_c3ps9wjygi9 |
| chat-long | llama.cpp@1 (9725a31) | Q4_K_M | 65.09tok/s | 3,506.5tok/s | 908ms | r_c3ps9wjygi9 |
| chat-short | llama.cpp@1 (9725a31) | Q4_K_M | 65.43tok/s | 865.3tok/s | 132ms | r_7mod38qsldj |
| chat-long | llama.cpp@1 (9725a31) | Q4_K_M | 65.10tok/s | 3,501.1tok/s | 909ms | r_7mod38qsldj |