A free, offline model
beat the cloud.
Gemma 4 vs Claude on a real code-tagging benchmark. April 2026.
TL;DR
The five-second version.
Gemma 4 26B MoE (Q8 + Think), running offline on a single RTX 5090, scored 8.4, within 0.1 of Claude Opus 4.6 (8.5) and ahead of Haiku 4.5 (7.9) and Sonnet 4.5 (7.2), at $0.00 per file.
It resolved 96% of the real import dependencies in the audit (more than Opus 4.6 at 91%), the single metric that matters most for a code-knowledge graph.
Opus billed $4.56 to tag 59 files in ten minutes ($0.077 each). The local model tagged 91 files in the same window for the price of electricity, about $0.05 an hour.
The 26B MoE (3.8B active params at full Q8 precision) beat the 31B Dense model (30.7B params crushed to Q4, 38% imports). Quantization, not parameter count, decided quality.
Zero parse errors on the Q8 runs, Apache 2.0 licensed, no API keys and no rate limits: the whole benchmark ran on one workstation.
Methodology
Nine models. Same source. Same prompt. Same ten-minute clock. Gemma 4 at four quantization levels with and without thinking, plus Claude Opus 4.6, Sonnet 4.5, and Haiku 4.5. Each tagger ran against 327 TypeScript files from the openclaw CLI subsystem.
#1 Best overall
#2 Quality ceiling
#3 Best speed-to-quality
Free local model outscored $0.64 Haiku and $0.92 Sonnet.
$0.00 per file · zero parse errors · single RTX 5090Measured head-to-head.
| #▲ | Model | Speed | tok/s | Cost | Logical | Notes |
|---|---|---|---|---|---|---|
1 Best overall | Gemma 4 26B MoE Q8_0 + Think Google · Local · Free · Apache 2.0 | 9.1 files/min 6,537 ms/file \u00B7 91 files | 142.1 551.1 effective | $0.00 | 8.4 Best tagger overall | Best import completeness of any model tested. Perfect restraint on trivial files. Near-lossless Q8 weights + explicit reasoning. 0 parse errors. |
2 Quality ceiling | Claude Opus 4.6 Anthropic | 5.9 files/min 10,104 ms/file \u00B7 59 files | 42.1 338.4 effective | $4.56 | 8.5 Deepest QA, impractical for bulk | Gold standard QA depth. Near-perfect imports. Mega-tag habit hurts embeddings. $0.077/file. Too slow and expensive for bulk tagging. |
3 Best hosted | Claude Haiku 4.5 Anthropic | 13.7 files/min 4,376 ms/file \u00B7 137 files | 84.6 724.6 effective | $0.64 | 7.9 Fast but drops imports | Best QA insight of any hosted model. Drops 40% of imports on complex files. Narrative inflation on trivial files. 1 parse error (0.7%). |
Ranking is the ratio of logical score to speed to cost, not any single metric. tok/s is the raw generation rate; effective (in gray) includes all tokens processed (input + output + thinking), which inflates the number for thinking-enabled models without producing more useful work.
Why 31B Dense loses to 26B MoE
31B Dense has 8x more active parameters per token than 26B MoE. It should be smarter. It isn't.
Same file, same prompt. The model with fewer active parameters found 2.5x more real dependencies.
Root cause
Quantization is the variable, not parameter count.
Each bar = VRAM used out of 32 GB. ~3.5 GB reserved for OS/CUDA/KV cache. Higher precision = better imports, but takes more memory.
You save 75% of VRAM but lose almost all weight granularity. Multiply that rounding error by 30.7 billion weights. The model still generates fluent text, but subtle patterns like import tracking disappear.
Same file, same prompt. More parameters, worse results. The 26B MoE activates only 3.8B parameters per token at high precision. The 31B Dense crams 30.7B into Q4 and loses the ability to track imports.
Deep dive by category.
Import completeness · 4-file audit
Per-file import audit
| Model | channel-auth (/8) | cli-utils (/2) | clawbot-cli (/3) | channels-cli (/10) | Overall |
|---|---|---|---|---|---|
| 26B Q8 Think | 8/8 | 2/2 | 3/3 | 9/10 | 96% |
| Opus 4.6 | 7/8 | 1/2 | 3/3 | 10/10 | 91% |
| 26B Q4 Think | 8/8 | 1/2 | 3/3 | -- | 92% |
| 26B Q8 | 7/8 | 1/2 | 3/3 | -- | 85% |
| Haiku 4.5 | 7/8 | 1/2 | 3/3 | 6/10 | 74% |
| 26B Q4 | 6/8 | 1/2 | 3/3 | -- | 77% |
| Sonnet 4.5 | 5/8 | 1/2 | 3/3 | -- | 69% |
| 31B Think | 3/8 | 1/2 | 3/3 | -- | 54% |
| 31B Dense | 3/8 | 1/2 | 1/3 | -- | 38% |
Ground truth: we read every source file, counted every real import, then checked connects_to. channels-cli.ts (10 imports) was the hardest test: Haiku dropped 4 presentation-layer imports, Opus got all 10, Q8 Think got 9.
Local vs Opus 4.6 API.
Gemma Q8 Think scored 8.4, closest to Opus (8.5). Select an API model to see what that quality would cost.
| Opus 4.6 (API) | Gemma Q8 Think (local) | |
|---|---|---|
| 10 minutes | 59 files · $4.54 | 91 files · $0.00 |
| 1 hour | 354 files · $27.26 | 546 files · $0.05 electricity |
Opus 4.6 pricing: $15/M input + $75/M output tokens. Local electricity: RTX 5090 at ~450W, $0.12/kWh US average.
Save $74,436 in year one.
Hardware pays for everything else too.$5,000 PC vs Opus 4.6 API.
| Value | |
|---|---|
| PC build cost | $5,000 |
| Opus 4.6 cost per file | $0.077 |
| Files to break even | 64,935 files |
| At 9.1 files/min | ~119 hours of tagging |
| Break even | ~15 days (8h/day) |
| After 1 year savings | $74,436 |
The PC pays for itself in ~15 days at 8h/day. After one year you've saved ~$74,436. The machine also does everything else a $5K workstation does.
What Opus would have cost
Test environment
Hardware used for all benchmarks.
| Component | Spec |
|---|---|
| GPU | NVIDIA GeForce RTX 5090 · 32 GB GDDR7 |
| CPU | AMD Ryzen 9 9950X3D · 16 cores · 32 threads · 4.3 GHz base |
| System RAM | G.SKILL DDR5-6000 · 64 GB (2× 32 GB) · CL26 |
| Storage | Samsung 9100 PRO · 4 TB · PCIe 5.0 x4 NVMe · 14,800 MB/s |
| Motherboard | Gigabyte X870E AORUS ELITE WiFi7 |
| OS | Windows 11 Home |
Common questions
Frequently asked.
Did the free local model actually beat Claude?
On the metric that matters most for a code graph, yes. Gemma 4 26B MoE (Q8 + Think) scored 8.4 overall against 8.5 for Opus 4.6, but found more real imports than any model tested (96% versus 91% for Opus) at $0.00 per file instead of $0.077.
Why did the smaller 26B model beat the larger 31B one?
Because quantization, not parameter count, drives quality. The 26B MoE activates only 3.8B params per token but keeps them near full precision (Q8); the 31B Dense has to be crushed to 4-bit (Q4) to fit in 32 GB of VRAM, which destroys the weight granularity needed to track imports. It found only 38% of them.
What hardware do I need to run this?
A single NVIDIA RTX 5090 with 32 GB of VRAM. Every benchmark ran on one desktop (RTX 5090, Ryzen 9 9950X3D, 64 GB of RAM) with no cloud, no API keys and no rate limits.
Is it really free, or are there hidden costs?
The model is Apache 2.0 and runs entirely offline, so there is no per-file or per-token charge: you pay only for electricity, about $0.05 an hour on the RTX 5090. At volume, a $5,000 workstation pays for itself in weeks against Opus at $0.077 per file.
So should I still use Claude for this?
For deep, one-off QA on a single file, Opus is still the quality ceiling (8.5). For tagging thousands of files it is too slow and too costly: the free local model matches its import accuracy at $0.00 and a fraction of the time.