Local inference · April 2026
Best LLM for
RTX 5090.
32 GB of GDDR7. Only 28.7 GB usable for weights. One model owns that budget: Gemma 4 27B MoE Q8.
TL;DR · Key takeaways
Best model for 32 GB, in five lines.
Gemma 4 26B MoE at Q8 is the best LLM for a 32 GB RTX 5090. It wins the single-GPU tier outright, and it's not close.
You get ~28.7 GB usable for weights, not the full 32. DWM, the CUDA context, and KV cache eat ~3.3 GB before a single weight loads.
Q8 is the floor. A smaller model at Q8 beats a bigger model at Q4 on every reasoning benchmark that matters: precision is worth more than parameter count.
The 26B MoE lands ~97% of the 31B dense's quality while activating just 3.8B parameters per token: dense-model output at MoE cost.
Keep thinking mode on. Native reasoning runs in one pass with no anchoring bias, and a low-active-param MoE benefits from it most.
Usable for weights
GB
After DWM, CUDA, and KV cache overhead
Advertised
GB
3.3 GB eaten before you load a single weight
VRAM Budget
Where your 32 GB actually goes.
| Component | VRAM | Why |
|---|---|---|
| Physical GDDR7 | 32.0 GB | What NVIDIA advertises |
| Visible to Windows | ~31.5 GB | GPU firmware and display engine reserve ~0.5 GB before the OS sees it |
| DWM (Desktop Window Manager) | -0.2 to -1.0 GB | Cannot be disabled on Windows 11. Varies by monitor count and resolution |
| CUDA context initialization | -0.2 to -0.4 GB | Allocated when Ollama loads the model |
| KV cache (inference) | -0.5 to -2.0 GB | Grows with context length. 8K context needs ~0.5 GB, 32K needs ~2 GB |
| Available for weights | 28.7 to 30.2 GB | This is your real budget |
The 28.7 GB floor assumes a single 4K monitor, 8K context, and a warm CUDA context. If you run two monitors or extend context to 32K, your budget drops to ~27 GB.
Quantization
Q8 is the floor for serious work.
This is the single most common mistake in local model setups. People see that a 70B model can be squeezed into 32 GB at Q4 and assume bigger is always better. It's not.
A 26B model at Q8 outperforms a 70B model at Q4 on every reasoning benchmark that matters. The parameters you load at full precision are worth more than the parameters you butcher to make them fit.
The rule
higher precision.
Always beats bigger model, lower precision
Q8 overhead
Quality loss vs FP16. Negligible.
31B Dense on 5090
Can the bigger model fit?
The 31B dense scores 11 ELO higher. But it needs aggressive quantization to fit.
| Quant | Size | Fits? | Notes |
|---|---|---|---|
| FP16 | ~62 GB | No | Need 2x GPUs. Lossless but impractical on consumer hardware |
| Q8_0 | ~31 GB | No | Fills all VRAM. Zero room for KV cache: inference will OOM |
| Q6_K | ~24 GB | Yes | Fits with headroom but loses quality from quantization |
| Q5_K_M | ~21 GB | Yes | Comfortable fit. Measurable quality loss on reasoning tasks |
| Q4_K_M | ~18 GB | Yes | Tons of headroom. Significant quality drop |
What actually fits
Three models. One GPU. Only one wins.
The tradeoff
The quant loss exceeds the parameter gain. 31B at Q6_K scores worse than 26B MoE at Q8.
Head to head
31B Dense vs 26B MoE at full quality.
| Benchmark | 31B Dense (Q8) | 26B MoE (Q8) | Gap |
|---|---|---|---|
| Arena ELO | 1452 | 1441 | 11 pts |
| AIME 2026 (math) | 89.2% | 88.3% | 0.9% |
| LiveCodeBench v6 (coding) | 80.0% | 77.1% | 2.9% |
| GPQA Diamond (science) | 84.3% | 82.3% | 2.0% |
| MMLU Pro | 85.2% | 82.6% | 2.6% |
The 26B MoE delivers ~97% of the 31B dense quality. The biggest gap is coding at 2.9%. Everything else is within 2 points. For practical work, the gap is negligible.
Architecture
MoE is the cheat code for consumer GPUs.
26B total parameters, but only 3.8B active per token. 128 experts, 2 active at a time, a 1.6% activation ratio. You get dense-model quality with a fraction of the compute cost per token.
Google trained this with the Gemini pipeline and distilled the routing decisions. Nobody else has MoE this efficient at consumer scale.
Total parameters
128 experts, 2 active per token
Active per token
1.6% activation ratio
Model fitting
Everything that fits on a single 5090.
Q8 quantization. 28.7-30.2 GB weight budget. Here's what makes the cut.
| Model | Quant | Weight size | Headroom | ELO | Notes |
|---|---|---|---|---|---|
| Gemma 4 26B MoE | Q8_0 | ~26 GB | ~4-6 GB | 1441 | Winner. MoE gives dense-quality at MoE efficiency |
| Qwen 3.5 27B | Q8_0 | ~27 GB | ~3-5 GB | ~1430 | Fits but tighter. No published coding benchmarks |
| Mistral Small 3.1 24B | Q8_0 | ~24 GB | ~6-8 GB | ~1380 | Comfortable fit but -61 ELO. Over a year old |
| Gemma 4 31B Dense | Q8_0 | ~31 GB | 0 GB | 1452 | Won’t work: no KV cache room at Q8 |
| Gemma 4 31B Dense | Q6_K | ~24 GB | ~6-8 GB | <1452 | Fits but quant loss > parameter gain over 26B Q8 |
Thinking mode
Always keep it on. It's not optional.
How it works
Reasons before committing to answer
Performance cost
Single forward pass, longer output
Why it matters for MoE
Smaller active params benefit more from explicit reasoning
Think mode activates via think: true in the Ollama API. The model generates reasoning traces inside <think>...</think> tokens before producing the final answer. One generation, longer output.
Native thinking reasons before generating, so there's no anchoring bias. A DIY "check your work" loop generates first, then tries to validate, but it's already anchored to its first answer. Native thinking also runs in one pass, while DIY loops need 2-3x full inference passes.
With only 3.8B active parameters, explicit reasoning helps the model enumerate imports, function signatures, and call patterns before committing. Without it, the model pattern-matches and guesses.
Gemma 4 26B MoE Q8 + Think. The king of single-GPU inference.
97% of 31B dense quality. 100% GPU utilization. Zero OOM.Verdict
The podium for 32 GB.
#1
#2
#3
The 31B dense would need Q6_K to fit, at which point you've lost more quality from quantization than you gained from the extra parameters. Qwen 3.5 27B is competitive on ELO but has no published coding benchmarks. Mistral Small 3.1 is over a year old and 61 ELO behind.
The next meaningful upgrade requires 2x GPUs (Qwen 3.5 72B, ~$4K) for +19 ELO. Below that threshold, Gemma 4 26B MoE is the undisputed king of the single-GPU tier, and it's not close.
FAQ · Common questions
Questions, answered.
What is the best LLM for a 32 GB RTX 5090?
Gemma 4 26B MoE at Q8. It loads in ~26 GB with room left for KV cache, scores 1441 Arena ELO, and delivers roughly 97% of the 31B dense model's quality while activating only 3.8B parameters per token.
Can I run a 70B model or the Gemma 4 31B dense on a single 5090?
Only by quantizing below Q8, and it isn't worth it. The 31B dense needs Q6_K to fit, and the quality lost to quantization exceeds the gain from the extra parameters. It ends up scoring below the 26B MoE at Q8.
How much of the 32 GB can I actually use for model weights?
About 28.7 to 30.2 GB. Windows sees ~31.5 GB, then DWM, the CUDA context, and the KV cache take ~3.3 GB before the first weight loads. Budget for 28.7 GB on a single 4K monitor at 8K context.
Do I need to keep thinking mode on?
Yes. Native thinking reasons before it answers in a single forward pass, so there's no anchoring bias, and it matters most for an MoE with only 3.8B active parameters: explicit reasoning lets it enumerate imports and call patterns instead of guessing.
What's the fastest way to give a local model access to my files?
Point your agent at Interlinked Files: a live, whole-machine index that returns a filename in ~85 µs and a full-text code search in ~7 to 9 ms, where ripgrep takes 93.8 s cold, across 4.47M files on a Ryzen 9 9950X3D. For an agent, that collapses a 6m57s, 71-call retrieval into a single 16 ms lookup while it idles in ~44 MB of RAM. Everything local is free, forever: the app, whole-machine search, MCP. Hosting starts at $5.99/mo.