Nemotron 3.5 Bolt 74B fits 24GB: 3 GPUs tested
Published 2026-10-16
Nemotron 3.5 Bolt 74B-A9B fits in 24GB for single-user use. With NVIDIA's official 2-bit GGUF (22.0 GB) on llama.cpp 620d2225 we measured 107.3 tok/s on an RTX 4090, 87.8 on an RTX 3090 and 154.7 on an RTX 5090 at 4K context — 2.06× Qwen3.8-27B Q4_K_M on the same 5090 and build. Ollama 0.40.4 cannot load it yet. Full table, longer-context rows and FAQ: /m/nemotron-3-5-bolt-74b-a9b.
Decode figures are suite-v1 chat-short, one run per config (not a median). Time to first token uses suite-v1 chat-long: this tokenizer counted 3,249 prompt tokens (suite target 4,096). Peak VRAM on the pods was about 22.0 of 24.6 GB (4090), 21.8 of 24.6 GB (3090) and 22.3 of 32.6 GB (5090). Concurrent-decode at 4K/8K on the 24GB cards logged Context size has been exceeded retries.
Measured at 4K context
| Reported hardware | Model · quant | Backend · workload | Decode (measured) | TTFT (measured) | Run |
|---|---|---|---|---|---|
| RTX 3090 (24GB) | Nemotron-3.5-Bolt-74B-A9B-24GB quant not reported | llama.cpp 0.7.0-dev (build 11669, commit 620d2225c) chat-short · RTX 3090 24GB · 87.8 tok/s · TTFT 2.75 s (3,249-tok prompt) · prefill 1,183.4 tok/s | 87.75tok/s | 462ms | r_-sifd5otbpy 2026-10-15 |
| RTX 4090 (24GB) | Nemotron-3.5-Bolt-74B-A9B-24GB quant not reported | llama.cpp 0.7.0-dev (build 11669, commit 620d2225c) chat-short · RTX 4090 24GB · 107.3 tok/s · TTFT 1.79 s · prefill 1,814.7 tok/s | 107.3tok/s | 469ms | r_vkjsinuhlyl 2026-10-15 |
| RTX 5090 (32GB) | Nemotron-3.5-Bolt-74B-A9B-24GB quant not reported | llama.cpp 0.7.0-dev (build 11669, commit 620d2225c) chat-short · RTX 5090 32GB · 154.7 tok/s · TTFT 1.24 s · prefill 2,623.3 tok/s | 154.7tok/s | 340ms | r_z-l1yelv_67 2026-10-15 |
Same 5090, vs Qwen3.8-27B
| Reported hardware | Model · quant | Backend · workload | Decode (measured) | TTFT (measured) | Run |
|---|---|---|---|---|---|
| RTX 5090 (32GB) | Nemotron-3.5-Bolt-74B-A9B-24GB quant not reported | llama.cpp 0.7.0-dev (build 11669, commit 620d2225c) chat-short · Nemotron 3.5 Bolt 74B-A9B 2-bit · 154.7 tok/s | 154.7tok/s | 340ms | r_z-l1yelv_67 2026-10-15 |
| RTX 5090 (32GB) | Qwen3.8-27B Q4_K_M | llama.cpp 0.7.0-dev (build 11669, commit 620d2225c) chat-short · Qwen3.8-27B Q4_K_M · 75.2 tok/s | 75.19tok/s | 273ms | r_qfrb2zmyk0k 2026-10-15 |
Related: model page · Qwen3.8-27B · RTX 4090 · RTX 3090 · RTX 5090 · VRAM fit · why is my local LLM slow