Skip to content
llm-speed
Leaderboard/hardware/rtx-4090

RTX 4090 (48GB) LLM benchmark

The fastest LLM measured on the RTX 4090 (48GB) is gemma3 at 195.0 decode tok/s via ollama (signed run). Across 56 reproducible runs on 8 models, this page lists decode tok/s, prefill, and TTFT for each, every number linking to the run it came from.

Fastest known config on RTX 4090 (48GB)

195.0 decode tok/s

gemma3 via ollama (Q4_K_M). see full run

glm-4.7-flash

WorkloadBackendQuantdecode tok/sprefill tok/sTTFTRun
chat-shortollama@0.31.1Q4_K_Mno datano datano datar_o636l3cc-rr
chat-longollama@0.31.1Q4_K_Mno datano datano datar_o636l3cc-rr
concurrent-decodeollama@0.31.1Q4_K_Mno datano datano datar_o636l3cc-rr
agent-traceollama@0.31.1Q4_K_M129.9tok/s2,491.1tok/s1,262msr_o636l3cc-rr

qwen3.6

WorkloadBackendQuantdecode tok/sprefill tok/sTTFTRun
chat-shortollama@0.31.1Q4_K_Mno datano datano datar_h_659oy695r
chat-longollama@0.31.1Q4_K_Mno datano datano datar_h_659oy695r
concurrent-decodeollama@0.31.1Q4_K_Mno datano datano datar_h_659oy695r
agent-traceollama@0.31.1Q4_K_M44.18tok/s1,910.5tok/s1,808msr_h_659oy695r

gemma3

WorkloadBackendQuantdecode tok/sprefill tok/sTTFTRun
chat-shortollama@0.31.1Q4_K_M46.95tok/s144.5tok/s817msr_x23y_sg24pm
chat-longollama@0.31.1Q4_K_M45.25tok/s1,594.1tok/s1,998msr_x23y_sg24pm
concurrent-decodeollama@0.31.1Q4_K_M46.17tok/sno datano datar_x23y_sg24pm
agent-traceollama@0.31.1Q4_K_M45.33tok/s2,556.8tok/s956msr_x23y_sg24pm
chat-shortollama@0.31.1Q4_K_M92.65tok/s171.4tok/s689msr_3kmrc135e0e
chat-longollama@0.31.1Q4_K_M87.60tok/s2,206.4tok/s1,444msr_3kmrc135e0e
concurrent-decodeollama@0.31.1Q4_K_M90.43tok/sno datano datar_3kmrc135e0e
agent-traceollama@0.31.1Q4_K_M88.53tok/s3,123.9tok/s815msr_3kmrc135e0e
chat-shortollama@0.31.1Q4_K_M195.0tok/s167.2tok/s706msr_dlanfbgym0h
chat-longollama@0.31.1Q4_K_M187.6tok/s3,254.1tok/s979msr_dlanfbgym0h
concurrent-decodeollama@0.31.1Q4_K_M193.1tok/sno datano datar_dlanfbgym0h
agent-traceollama@0.31.1Q4_K_M190.0tok/s3,543.4tok/s715msr_dlanfbgym0h

r1

WorkloadBackendQuantdecode tok/sprefill tok/sTTFTRun
chat-shortollama@0.31.1Q4_K_Mno datano datano datar_zrjj-pj93q8
chat-longollama@0.31.1Q4_K_M81.09tok/s240.4tok/s13,070msr_zrjj-pj93q8
concurrent-decodeollama@0.31.1Q4_K_Mno datano datano datar_zrjj-pj93q8
agent-traceollama@0.31.1Q4_K_Mno datano datano datar_zrjj-pj93q8
chat-shortollama@0.31.1Q4_K_Mno datano datano datar_fg77v2hhohb
chat-longollama@0.31.1Q4_K_M131.9tok/s686.3tok/s4,578msr_fg77v2hhohb
concurrent-decodeollama@0.31.1Q4_K_Mno datano datano datar_fg77v2hhohb
agent-traceollama@0.31.1Q4_K_M133.8tok/s8,852.4tok/s366msr_fg77v2hhohb
chat-shortollama@0.31.1Q4_K_Mno datano datano datar_l2ck67ls40i
chat-longollama@0.31.1Q4_K_M3.76tok/s14.28tok/s219,986msr_l2ck67ls40i
concurrent-decodeollama@0.31.1Q4_K_Mno datano datano datar_l2ck67ls40i
agent-traceollama@0.31.1Q4_K_M3.79tok/s3,515.1tok/s971msr_l2ck67ls40i
chat-shortollama@0.31.1Q4_K_Mno datano datano datar_2r1p3w9ps7c
chat-longollama@0.31.1Q4_K_M131.0tok/s599.9tok/s5,238msr_2r1p3w9ps7c
concurrent-decodeollama@0.31.1Q4_K_Mno datano datano datar_2r1p3w9ps7c
agent-traceollama@0.31.1Q4_K_M132.4tok/s9,612.8tok/s337msr_2r1p3w9ps7c

qwen3-coder

WorkloadBackendQuantdecode tok/sprefill tok/sTTFTRun
chat-shortollama@0.31.1Q4_K_M145.2tok/s195.6tok/s562msr_wjq32z47vlp
chat-longollama@0.31.1Q4_K_M165.5tok/s3,527.2tok/s892msr_wjq32z47vlp
concurrent-decodeollama@0.31.1Q4_K_M179.9tok/sno datano datar_wjq32z47vlp
agent-traceollama@0.31.1Q4_K_Mno datano datano datar_wjq32z47vlp

gpt-oss

WorkloadBackendQuantdecode tok/sprefill tok/sTTFTRun
chat-shortollama@0.31.1MXFP4no datano datano datar_iu2sfa9ykvw
chat-longollama@0.31.1MXFP4141.7tok/s631.5tok/s5,048msr_iu2sfa9ykvw
concurrent-decodeollama@0.31.1MXFP4no datano datano datar_iu2sfa9ykvw
agent-traceollama@0.31.1MXFP4no datano datano datar_iu2sfa9ykvw

qwen2.5-coder

WorkloadBackendQuantdecode tok/sprefill tok/sTTFTRun
chat-shortollama@0.31.1Q4_K_M89.45tok/s466.6tok/s281msr_73tnfdueq2h
chat-longollama@0.31.1Q4_K_M83.84tok/s3,572.3tok/s887msr_73tnfdueq2h
concurrent-decodeollama@0.31.1Q4_K_M88.63tok/sno datano datar_73tnfdueq2h
agent-traceollama@0.31.1Q4_K_M86.21tok/s5,298.4tok/s395msr_73tnfdueq2h
chat-shortollama@0.31.1Q4_K_M159.4tok/s452.0tok/s290msr_mv8n8k9wu1e
chat-longollama@0.31.1Q4_K_M154.7tok/s5,998.7tok/s528msr_mv8n8k9wu1e
concurrent-decodeollama@0.31.1Q4_K_M161.1tok/sno datano datar_mv8n8k9wu1e
agent-traceollama@0.31.1Q4_K_M158.1tok/s6,421.1tok/s320msr_mv8n8k9wu1e

llama3.1

WorkloadBackendQuantdecode tok/sprefill tok/sTTFTRun
chat-shortollama@0.31.1Q4_K_M154.4tok/s328.0tok/s335msr_h1ub_1uxzdh
chat-longollama@0.31.1Q4_K_M146.3tok/s5,032.5tok/s623msr_h1ub_1uxzdh
concurrent-decodeollama@0.31.1Q4_K_M154.2tok/sno datano datar_h1ub_1uxzdh
agent-traceollama@0.31.1Q4_K_M149.4tok/s5,377.5tok/s387msr_h1ub_1uxzdh

Community folklore on RTX 4090 (48GB)

108 unverified claims extracted from Reddit/HN comments. Lower trust than signed runs above; every row links to the source.

  • communityconfidence 75%

    60.00tok/s — Llama 3.1 70b on RTX 4090 via ollama IQ2_XS

    our signed data: RTX 4090 · Llama 3.1 70b

    “Llama 3.1 70b at 60 tok/s on RTX 4090 (IQ2_XS) Setup GPU: 1 x RTX 4090 (24 GB VRAM) CPU: Xeon® E5-2695 v3 (16 cores) RAM: 64 GB RAM Running PyTorch 2.2.0 + CUDA 1”

    source: Reddit · u/grey-seagull · 2024-09-20

  • communityconfidence 75%

    60.00tok/s — Llama 3.1 70b on RTX 4090 via ollama IQ2_XS

    our signed data: RTX 4090 · Llama 3.1 70b

    “Llama 3.1 70b at 60 tok/s on RTX 4090 (IQ2_XS) Setup GPU: 1 x RTX 4090 (24 GB VRAM) CPU: Xeon® E5-2695 v3 (16 cores) RAM: 64 GB RAM Running PyTorch 2.2.0 + CUDA 1”

    source: Reddit · u/grey-seagull · 2024-09-20

  • communityconfidence 75%

    60.00tok/s — Llama 3.1 70b on RTX 4090 via ollama IQ2_XS

    our signed data: RTX 4090 · Llama 3.1 70b

    “Llama 3.1 70b at 60 tok/s on RTX 4090 (IQ2_XS) Setup GPU: 1 x RTX 4090 (24 GB VRAM) CPU: Xeon® E5-2695 v3 (16 cores) RAM: 64 GB RAM Running PyTorch 2.2.0 + CUDA 1”

    source: Reddit · u/grey-seagull · 2024-09-20

  • communityconfidence 75%

    92.40tok/s — Qwen3-8B on RTX 4090 via llama.cpp Q8_0

    our signed data: RTX 4090 · Qwen3-8B

    “tok/s` generation at `128` output tokens - `Q8_0`: about `9975 tok/s` prompt processing at `512` tokens, `9955 tok/s` at `1024`, and about `92.4 tok/s` generation at `128` output tokens Hardware / runtime for those numbers: - `RTX 4090` - `Ryzen 9 7900X` - `llama.cpp` build com…”

    source: Reddit · u/RiverRatt · 2026-03-23

  • communityconfidence 75%

    92.40tok/s — Qwen3-8B on RTX 4090 via llama.cpp Q8_0

    our signed data: RTX 4090 · Qwen3-8B

    “tok/s` generation at `128` output tokens - `Q8_0`: about `9975 tok/s` prompt processing at `512` tokens, `9955 tok/s` at `1024`, and about `92.4 tok/s` generation at `128` output tokens Hardware / runtime for those numbers: - `RTX 4090` - `Ryzen 9 7900X` - `llama.cpp` build com…”

    source: Reddit · u/RiverRatt · 2026-03-23

  • communityconfidence 75%

    548.9tok/s — Llama-3.1-70B on RTX 4090 via vllm FP8

    our signed data: RTX 4090 · Llama-3.1-70B

    “neuralmagic\_Meta-Llama-3.1-70B-Instruct-FP8-dynamic Avg generation throughput: \~29-30 tokens/s Avg prompt throughput: 548.9 tokens/s (4 GPUs, 4090, power limited to 325) (8x, 8x, 4x , 4x) 5950x taichi x570 vllm backend i didn't do the specific prompt to get the va”

    source: Reddit · u/I_can_see_threw_time · 2024-08-01

  • communityconfidence 75%

    30.00tok/s — Llama-3.1-70B on RTX 4090 via vllm FP8

    our signed data: RTX 4090 · Llama-3.1-70B

    “neuralmagic\_Meta-Llama-3.1-70B-Instruct-FP8-dynamic Avg generation throughput: \~29-30 tokens/s Avg prompt throughput: 548.9 tokens/s (4 GPUs, 4090, power limited to 325) (8x, 8x, 4x , 4x) 5950x taichi x570 vllm backend i di”

    source: Reddit · u/I_can_see_threw_time · 2024-08-01

  • communityconfidence 75%

    548.9tok/s — Llama-3.1-70B on RTX 4090 via vllm FP8

    our signed data: RTX 4090 · Llama-3.1-70B

    “neuralmagic\_Meta-Llama-3.1-70B-Instruct-FP8-dynamic Avg generation throughput: \~29-30 tokens/s Avg prompt throughput: 548.9 tokens/s (4 GPUs, 4090, power limited to 325) (8x, 8x, 4x , 4x) 5950x taichi x570 vllm backend i didn't do the specific prompt to get the va”

    source: Reddit · u/I_can_see_threw_time · 2024-08-01

  • communityconfidence 75%

    30.00tok/s — Llama-3.1-70B on RTX 4090 via vllm FP8

    our signed data: RTX 4090 · Llama-3.1-70B

    “neuralmagic\_Meta-Llama-3.1-70B-Instruct-FP8-dynamic Avg generation throughput: \~29-30 tokens/s Avg prompt throughput: 548.9 tokens/s (4 GPUs, 4090, power limited to 325) (8x, 8x, 4x , 4x) 5950x taichi x570 vllm backend i di”

    source: Reddit · u/I_can_see_threw_time · 2024-08-01

  • communityconfidence 70%

    140.0tok/s — Qwen3-Coder on RTX 4090 via llama.cpp

    our signed data: RTX 4090 · Qwen3-Coder

    “bled, KV cache at Q8) on my 4090. This is fully on the GPU, no offloading to CPU. Depending on context length I'm getting anywhere from 100-140 tokens/sec. If you wanted more context you'd have to offload some layers to CPU and it takes a massive hit (my recent post has some benc…”

    source: Reddit · u/ConversationNice3225 · 2025-08-05

See all 108 claims for RTX 4090 (48GB) →

Models measured on RTX 4090 (48GB)

Common questions about RTX 4090 (48GB)

Direct Q&A drawn from the runs above: fastest LLM, supported model classes, backend rankings, quantization guidance.

Read the RTX 4090 (48GB) FAQ →