Nemotron 3.5 Bolt 74B-A9B: how fast is it on a 24GB card?

Nemotron 3.5 Bolt 74B-A9B fits in 24GB for single-user use: with NVIDIA's official 2-bit GGUF (22.0 GB) on llama.cpp 620d2225 it ran at 107.3 tok/s on an RTX 4090 and 87.8 tok/s on an RTX 3090 at 4K context, and 154.7 tok/s on an RTX 5090 — 2.06× the 75.2 tok/s of Qwen3.8-27B Q4_K_M on the same 5090 and build (one run each).

Setup: NVIDIA official 2-bit GGUF (22.0 GB on disk), llama.cpp 620d2225 (0.7.0-dev build 11669), all layers on the GPU. Decode figures below are suite-v1 chat-short (one run per config, not a median). Time to first token and prefill are from suite-v1 chat-long: this tokenizer counted 3,249 prompt tokens (suite target 4,096).

Measured at 4K context

Longer context

Vs Qwen3.8-27B (same 5090, same build)

On the same RTX 5090 pod and llama.cpp build (620d2225) at 4K, Nemotron 3.5 Bolt 74B-A9B (2-bit, 22.0 GB) decoded 154.7 tok/s (r_z-l1yelv_67) vs 75.2 tok/s for dense Qwen3.8-27B Q4_K_M (r_qfrb2zmyk0k): 2.06× faster. Bolt reads only its ~9B active parameters per token from a 2-bit file, so less memory traffic per token than a 27B dense Q4 model. That explains speed, not quality; we don't measure quality.

24GB fit (single-user) and concurrent limit

For one user at a time the 2-bit file fits: peak VRAM was about 22.0 of 24.6 GB on the 4090 (all three context sizes we tried), about 21.8 of 24.6 GB on the 3090 at 8K, and about 22.3 of 32.6 GB on the 5090. Concurrent-decode at 4K and 8K on the 24GB cards logged Context size has been exceeded retries in the server log; the workload still reported ok, but multi-user headroom on 24GB is tighter than the single-stream numbers suggest.

Ollama

Ollama 0.40.4 cannot load Bolt yet: its bundled llama.cpp lacks support (same tensor-shape error as older llama.cpp). Use llama.cpp 620d2225 or later. See why is my local LLM slow.

FAQ

Does Nemotron 3.5 Bolt 74B fit in 24GB of VRAM? For single-user use, yes. NVIDIA's official 2-bit GGUF is 22.0 GB and the whole model stayed in VRAM with llama.cpp 620d2225 on both 24GB cards: RTX 4090 up to 16K context (105.7 tok/s chat-short, peak about 22.0 of 24.6 GB, r_qibhq5bkmgq) and RTX 3090 up to 8K (80.3 tok/s, peak about 21.8 of 24.6 GB, r_w3lond-s75t). Those are the largest contexts we ran, not proven limits. Concurrent-decode at 4K and 8K on the 24GB cards logged "Context size has been exceeded" retries while the workload still reported ok.

How fast is Nemotron 3.5 Bolt vs Qwen3.8-27B? About 2x faster on decode in our controlled test: 154.7 tok/s (r_z-l1yelv_67) vs 75.2 tok/s for Qwen3.8-27B Q4_K_M (r_qfrb2zmyk0k), same RTX 5090 pod, same llama.cpp build 620d2225, 4K context, one run each. Bolt reads only its ~9B active parameters per token from a 2-bit file, so less memory traffic per token than a 27B dense Q4 model. That explains speed, not quality; we don't measure quality.

Does Nemotron 3.5 Bolt work in Ollama? Not yet. On launch day Ollama 0.40.4 could not load Bolt because its bundled llama.cpp lacks support for the model (same tensor-shape error we saw on older llama.cpp). Use llama.cpp build 620d2225 or later, the first build that loads NVIDIA's 2-bit file.

Not measured yet

• Any Mac, RTX Spark, or Ollama tok/s (Ollama 0.40.4 cannot load it).
• 2-bit answer quality. We measure speed and memory only.
• Max context on 24GB beyond 16K (4090) / 8K (3090).

Related: day-one blog post · Qwen3.8-27B · RTX 4090 · RTX 3090 · RTX 5090 · VRAM fit · Qwen3.6-35B-A3B · MTP on 4090 · best GPU 2026