Can an RTX 3090 run Nemotron 3.5 Bolt 74B-A9B?
Yes: an RTX 3090 runs Nemotron 3.5 Bolt 74B-A9B at 87.8 tok/s at IQ2 (NVIDIA mixed 2-bit), measured on llama.cpp with 4K context in October 2026.
2 signed runs on llm-speed. Single-request decode speed in tokens per second; the short-chat workload unless noted. All runs were measured by llm-speed (@llmspeed). Runs measured by llm-speed are credited to @llmspeed.
Measured tok/s by quant and runtime
| Quant | Runtime | Build | Decode tok/s (best run) | Workload | Runs | Source |
|---|---|---|---|---|---|---|
| IQ2 (NVIDIA mixed 2-bit) | llama.cpp | commit 620d2225c | 87.8 | chat-short | 2 | Measured by llm-speed@llmspeed |
Fastest quant that fits: IQ2 (NVIDIA mixed 2-bit) on llama.cpp, 87.8 tok/s (measured by llm-speed; it fits because it ran on this hardware). Quants without a run are not listed.
VRAM used: 21.3 GiB at IQ2 (NVIDIA mixed 2-bit) (r_-sifd5otbpy) peak, 4K context, of 24.0 GiB (nvidia-smi peak, whole card). Process VRAM (llama-server, nvidia-smi): 21.1 GiB at IQ2 (NVIDIA mixed 2-bit).
Every measured row
| Run | Date (UTC) | Workload | Quant | Runtime / build | Prompt tokens | Output tokens | Decode tok/s | Prefill tok/s | TTFT ms | VRAM GiB | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|
| r_-sifd5otbpy | 2026-10-15 | chat-short | IQ2 (NVIDIA mixed 2-bit) | llama.cpp · commit 620d2225c | 120 | 256 | 87.8 | 259.7 | 462 | 21.3 peak | Measured by llm-speed@llmspeed |
| r_w3lond-s75t | 2026-10-15 | chat-short | IQ2 (NVIDIA mixed 2-bit) | llama.cpp · commit 620d2225c | 120 | 256 | 80.3 | 264.3 | 454 | 21.3 peak | Measured by llm-speed@llmspeed |
| r_-sifd5otbpy | 2026-10-15 | chat-long | IQ2 (NVIDIA mixed 2-bit) | llama.cpp · commit 620d2225c | 3249 | 847 | 87.5 | 1183.4 | 2746 | 21.3 peak | Measured by llm-speed@llmspeed |
| r_w3lond-s75t | 2026-10-15 | chat-long | IQ2 (NVIDIA mixed 2-bit) | llama.cpp · commit 620d2225c | 3249 | 1024 | 80.4 | 1072.2 | 3030 | 21.3 peak | Measured by llm-speed@llmspeed |
| r_w3lond-s75t | 2026-10-15 | agent-trace | IQ2 (NVIDIA mixed 2-bit) | llama.cpp · commit 620d2225c | 3449 | 840 | 79.3 | 2109.3 | 944 | 21.3 peak | Measured by llm-speed@llmspeed |
| r_-sifd5otbpy | 2026-10-15 | agent-trace | IQ2 (NVIDIA mixed 2-bit) | llama.cpp · commit 620d2225c | 3449 | 840 | 78.9 | 2095.0 | 895 | 21.3 peak | Measured by llm-speed@llmspeed |
| r_-sifd5otbpy | 2026-10-15 | concurrent-decode | IQ2 (NVIDIA mixed 2-bit) | llama.cpp · commit 620d2225c | 1024 | 256 | 87.8 | – | – | 21.3 peak | Measured by llm-speed@llmspeed |
| r_w3lond-s75t | 2026-10-15 | concurrent-decode | IQ2 (NVIDIA mixed 2-bit) | llama.cpp · commit 620d2225c | 1024 | 256 | 80.4 | – | – | 21.3 peak | Measured by llm-speed@llmspeed |
Questions
Can an RTX 3090 run Nemotron 3.5 Bolt 74B-A9B?
Yes: an RTX 3090 runs Nemotron 3.5 Bolt 74B-A9B at 87.8 tok/s at IQ2 (NVIDIA mixed 2-bit), measured on llama.cpp with 4K context in October 2026.
How many tokens per second does Nemotron 3.5 Bolt 74B-A9B get on an RTX 3090?
Decode speed measured by llm-speed, by quantization and runtime: IQ2 (NVIDIA mixed 2-bit) on llama.cpp: 87.8 tok/s. Based on 2 runs measured by llm-speed; single-request chat-short unless noted.
What is the fastest quant of Nemotron 3.5 Bolt 74B-A9B that fits on an RTX 3090?
IQ2 (NVIDIA mixed 2-bit) on llama.cpp was the fastest measured setting at 87.8 tok/s; it fit because it ran. Other quants are listed only where a run measured them.
How much VRAM does Nemotron 3.5 Bolt 74B-A9B use on an RTX 3090?
nvidia-smi recorded a peak of 21.3 GiB of 24.0 GiB at IQ2 (NVIDIA mixed 2-bit) with 4K context (run r_-sifd5otbpy); process VRAM for llama-server was 21.1 GiB.