Why Long Prompts Are Slow on a Mac: Prefill & TTFT Fixed

On our 48 GiB M5 Max, Qwen3.8-27B (MLX 4-bit) took a median 8.17 s to show its first token for an 840-token prompt and 113.21 s for 12,263 tokens. Decode stayed at about 13–14 tok/s either way. Reusing a prepared prefix cut the wait to 0.87 s. (Measured, llm-speed exploratory study, 16 Sep 2026.)

These numbers come from our Sep 16 M5 Max study, an exploratory native-MLX study, not signed suite-v1 leaderboard runs. Download the samples (CSV) and the medians and ranges (JSON).

Measured: first-token wait vs prompt length

Setup: MLX 0.32.1 / mlx-lm 0.31.3, mlx-community/Qwen3.8-27B-4bit pinned to commit 3e6447f, batch 1, thinking off, greedy decoding, 2,048-token prefill chunks, 256-token output cap, no KV quantization, no draft model or MTP. Shared workstation (free memory never below 23%, no additional swap). Preparing the 12,199-token prefix for the last row took 120.80 s (118.65–134.77) and is excluded from that row's request timing. Peak MLX allocation is not total system memory.

Two speeds, not one

A local model works in two phases. Prefill reads your whole prompt before anything appears; it is mostly compute-bound. Decode writes the answer one token at a time; it is mostly memory-bandwidth-bound. On our M5 Max, decode barely moved (13.70, 13.62 and 13.34 tok/s), while the wait for the first token grew about 14x from 840 to 12,263 tokens. Dividing prompt tokens by median TTFT gives about 103, 114 and 108 tok/s of prefill for the three fresh rows (calculated from measured medians, not a separately measured prefill rate). More on the difference in decode vs prefill tok/s explained.

Chat vs agents and RAG

A short chat prompt waits seconds. A 12K-token prompt meant about 2 minutes before any output in our test (measured). Coding agents and RAG pipelines resend long contexts on every turn, so this is the wall people hit: one user trying Qwen3.8-27B on a 32GB M1 Max wrote that “a ten minute prefill would kill a harness” (user report).

What cuts the wait

1. Prefix (prompt) caching. Measured. With 12,199 of 12,263 tokens already prepared, the first token arrived in 0.87 s instead of 113.21 s. Building that cache took 120.80 s on its own, so it only pays off when the same prefix is reused across requests.
2. Keep the system prompt and tool definitions stable so the cached prefix actually matches. Anything that changes early in the prompt forces a fresh prefill from that point.
3. Send less context: trim files, summarise or compact old turns.
4. Prefill chunk size, engine choice and speculative decoding: not measured by us. Users report llama.cpp prefilling faster while MLX decodes faster (example), that MLX is not always faster end to end (example), and that larger prefill steps help (example). Treat those as user reports; their numbers are not ours.

Will a bigger Mac fix it?

We don't know yet. We have no M3 Ultra, M4 Max or M5 Ultra first-token measurements at these prompt lengths; our M3 Ultra runs are short-chat decode only. For what we have measured on each Mac, see the best Mac for LLMs guide.

What we haven't measured

• The same model and settings on M3 Ultra, M4 Max or M5 Ultra.
• An NVIDIA comparison at the same prompt lengths. Our RTX 5090 Qwen3.8-27B runs are short-chat only, with a different engine and quant.
• llama.cpp vs MLX prefill on the same M5 Max; prefill chunk size (2,048 vs 8,192); KV quantization.
• 32K, 64K or 128K prompts. People ask about 100K-token agents; we stop at 12K.
• Any effect of MTP or draft models on first-token time.
• Tighter repeatability: one shared machine, 3 repetitions, wide min–max ranges.

Full setup, limits and reproduction files are in the Sep 16 M5 Max study. Check memory at longer context with the memory-fit estimator, see all Qwen3.8-27B results, or read why is my local LLM slow? Terms are defined in the glossary.

FAQ

Why is my Mac slow before the first token appears? Before writing anything, the model has to process the whole prompt (prefill), and that time grows with prompt length. On our M5 Max with Qwen3.8-27B, it went from 8.17 s at 840 tokens to 113.21 s at 12,263 tokens (measured medians).

Does a longer prompt make generation slower too? Barely, in our test: stream decode was 13.70, 13.62 and 13.34 tok/s at 840, 3,156 and 12,263 tokens (measured medians).

Does prompt caching help on Apple Silicon? Yes, when the prefix is reused: time to first token fell to 0.87 s. Building the 12,199-token prefix took 120.80 s separately (measured).

Is MLX or llama.cpp faster at prefill on a Mac? We haven't measured this. Users report llama.cpp prefilling faster while MLX decodes faster (user report).

How much memory did 12K context use? Peak MLX allocation was 18.48 GiB at 12,263 tokens vs 15.67 GiB at 840 (measured; not total system RAM).

Is a Mac good for long-context coding agents? It works, but long fresh prompts mean long waits (about 2 minutes at 12K tokens on our M5 Max), so caching matters more than decode speed.