Which qwen Actually Fits a 16GB GPU

The last time I benchmarked this RTX 5060 Ti 16GB, the question was which gemma4 12B to pull. This round is qwen, on a freshly updated driver and Ollama, and the question is whether the current generation runs on a 16GB card at all.

It runs, but no tag of it fits in VRAM. Ollama puts part of the model on the CPU, and the speed depends mostly on how many layers stay on the GPU. The default tag gets 6.6 tokens/sec, and the same weights with two Modelfile lines get 11.8. For a qwen that fits entirely on the card I had to go back to qwen3.5 9B, which does 38 to 67 depending on the build.

Why a 16GB card

GPU prices keep going up, and 16GB is where I think the trade between price and what you can run is best. The 5060 Ti is the cheapest 16GB card in the 50-series. A 5090 has twice the VRAM and ran gemma4 12B about 2.7 times faster in the last round, but it costs several times as much, and most people who want to run a model at home are not going to buy one.

So what runs on this card is closer to what most people can run. That is getting harder to take for granted. qwen3.5 shipped a 9B that fits here with room left over. qwen3.6 and qwen3.8 start at 17 GB, which is sized for a 24GB card. If new model families keep skipping the small sizes, the 16GB owner is left choosing between an older model that fits and a newer one that does not, and that choice is what the rest of this post measures.

Test setup

Same machine and harness as the gemma4 post, with a newer driver and Ollama.

• GPU: NVIDIA GeForce RTX 5060 Ti 16GB, driver 617.42, also driving the Windows desktop (about 1.2 GB of VRAM)
• CPU and RAM: i7-8700K, 64 GB DDR4-2400
• Software: Ollama 0.40.1 in Docker, on WSL2
• Server flags: flash attention, q8_0 KV cache, keep-alive set to never unload
• Benchmark: num_ctx=8192, num_batch=1024, warmup plus 10 iterations over a fixed prompt set, in both --think and --no-think

Throughput is generation tokens per second averaged over the 10 iterations. The harness and every raw result are on GitHub.

Nothing in qwen3.8 fits

Every qwen3.8 tag on the Ollama library is the 27B. The smallest one that runs on CUDA is the Q4_K_M build: 16.8 GB of weights plus a 0.9 GB vision projector, before the KV cache and compute buffers. The rest are bigger or MLX-only.

The latest tag was updated a week before this run, so I checked whether a smaller build had appeared. It had not. The weights, projector, and parameters are byte-identical to last month’s build, and the only change is new metadata naming the runner.

So on 16GB, Ollama splits the model. Some layers go to the GPU and the rest run on the CPU, and the CPU side sets the pace.

Build	Speculative decoding	Layers on GPU	Tokens/sec (no-think)	Tokens/sec (think)
qwen3.8:27b-q4_K_M	off	54 of 66	6.0	6.0
qwen3.8:27b (the default)	depth 4	47 of 66	6.6	6.6
depth 2 Modelfile	depth 2	49 of 66	7.8	7.5
depth 2, pinned to 56 layers	depth 2	56 of 66	11.8	11.6

Six tokens per second is slow enough that I would only use it for jobs I can leave running. During these runs the GPU sits at about 22% utilization and 42 watts of a 180 watt limit, while six CPU cores are pegged.

The default tag is the wrong one

qwen3.8:27b ships with Multi-Token Prediction turned on. MTP is speculative decoding built into the model: a small head drafts a few tokens ahead and the full model verifies them in one pass. On a card where the model fits, that speed costs almost nothing.

Here it costs layers. MTP needs VRAM for the draft head and its context, and on a full card that VRAM comes out of the layers Ollama can place on the GPU. The default build keeps 47 where the plain Q4 keeps 54. Moving seven layers to the CPU cancels most of what speculation gains, and the net is 10%.

The shipped draft depth of 4 makes it worse. Acceptance falls off fast by position (77%, 54%, 41%, 31%), so the third and fourth drafted tokens mostly buy a wasted verify. Dropping to depth 2 is a two-line Modelfile, and it is 18% faster than the default while keeping two more layers on the GPU.

Pinning the layers adds another 51%

The bigger win is taking layer placement away from Ollama. Its auto-fit is conservative: it targets nearly 2 GB of free VRAM and stops at 49 layers for the depth 2 build. Setting num_gpu makes the auto-fit stand down.

FROM qwen3.8:27b
PARAMETER draft_num_predict 2
PARAMETER num_gpu 56

Pinned at 56 layers, the same weights run at 11.8 tokens/sec. That is 51% over the auto-fit and 79% over the default tag.

The room for seven more layers comes from the vision encoder. With num_gpu set, Ollama moves it to the CPU, which frees about 1.1 GB. Text generation is unaffected. Image input will encode on the CPU, and I have not timed it.

The pinned build peaks at 15.65 GB of 16.3, so there is about 660 MB of slack. Push further and the Windows driver can spill CUDA allocations into system RAM instead of failing, which would be slower than the CPU path and much harder to spot.

The last qwen that fits

For a qwen that sits entirely in VRAM, I had to go back two generations. qwen3.6 bottoms out at a 17 GB 27B, and there is no qwen3.7 on the library. qwen3.5 is the last family with small models, and the 9B is the largest one that fits a 16GB card.

Four builds of it run on CUDA, and all four stay 100% on the GPU.

Build	Quant	Speculative decoding	Peak VRAM	Tokens/sec (no-think)	Tokens/sec (think)
qwen3.5:9b	Q4_K_M	off	7.7 GB	55.2	55.6
qwen3.5:9b-mtp-q4_K_M	Q4_K_M	depth 2	9.3 GB	65.1	67.2
qwen3.5:9b-q8_0	Q8_0	off	10.9 GB	38.2	38.2
qwen3.5:9b-mtp-q8_0	Q8_0	depth 2	12.6 GB	53.2	56.0

Peak VRAM includes the desktop’s 1.2 GB.

MTP makes Q8 as fast as Q4

With VRAM to spare, speculative decoding costs no layers. The MTP builds carry a separate 2.4 GB draft model, about 1.6 GB of extra VRAM in practice, and they are faster in every row: 18 to 21% at Q4 and 39 to 47% at Q8.

Q8 gains twice as much because it starts further behind. Without MTP, Q8 costs 31% of throughput against Q4, the same bandwidth-bound behavior this card shows on every dense model: it has 9.5 GB of weights to stream per token instead of 5.6. The draft model costs the same either way, but each token it saves is a more expensive token at Q8, so the same acceptance rate buys twice the speedup.

That puts qwen3.5:9b-mtp-q8_0 at 53 to 56 tokens/sec against 55 for plain Q4. It runs near-full-precision weights at the speed of the 4-bit build and leaves 3.7 GB of VRAM free.

Thinking does not change throughput, but qwen3.5 spends about three quarters of its output on it. The same prompts produce four to five times as many tokens and take that much longer to answer.

The upgrade made it slower

I updated the driver and Ollama before this round, and the three auto-fit qwen3.8 builds came in 6 to 13% below last month. Finding out why took longer than the benchmarks.

To separate the causes I ran the old Ollama in a second container against the same model files. The driver came out fine: with Ollama held constant, gemma4 12B produced identical output 3% faster on driver 617.42 than on 616.92. Ollama 0.40.1 is about 6% slower than 0.33.2 on that same pure-GPU model, 38.6 against 41.0 tokens/sec.

The CPU-offloaded path lost 11% with Ollama held constant, at the same output, layer split, and draft acceptance. I have not isolated that one. It is either the driver’s CPU-to-GPU handoff or something on the Windows side that changed in a month.

The last cause was outside the stack. A background tray app on the Windows desktop was holding half a CPU core and 10% of the idle GPU. It cost another 5 to 7%, and I spent a while treating it as a driver regression because WSL cannot see Windows processes and nothing in the Linux tooling showed it. I now check the Windows side before a run.

Bottom line

For qwen on a 16GB GPU, in priority order:

1. qwen3.5:9b-mtp-q8_0 is the default. 53 to 56 tokens/sec, sub-second first token, Q8 weights, all on the GPU.
2. qwen3.5:9b-mtp-q4_K_M when speed or context headroom matters more. 65 to 67 tokens/sec in 8 GB.
3. qwen3.8 pinned at 56 layers when the task needs the 27B and 12 tokens/sec is acceptable. Text only.
4. Do not run the default qwen3.8:27b tag on this card. It keeps the fewest layers on the GPU, and plain Q4 is the only build here that runs slower.

I only measured speed. A 27B is a stronger model than a 9B, so choosing between the first and third options means trading capability against four to five times the throughput.

The gemma4 round found acceleration features that would not run on Linux. This round found one that runs and has to be paid for in VRAM. On a model that fits, MTP is worth 18 to 47%. On a model that does not, it takes VRAM from the layers it is supposed to speed up, and at the default settings most of the gain is lost.

The harness, the Modelfiles, and every raw result are in bschwabQ/ollama-load-test.