Methodology -- April 2026
The benchmark
trap.
The #1 coding model might not actually be #1. Published benchmarks don't normalize for thinking mode: some models are tested with reasoning on, others with it off, and one model gets worse at coding when thinking is enabled.
Models with fair benchmarks
same mode published as benchmarked
Misleading comparisons
handicapped, inflated, or mode-switching
The trap
Three ways thinking mode breaks comparisons.
Always-on thinking inflates scores. MiniMax M2.5 runs interleaved thinking on every token: baked in, no off switch. Comparing its 80.2% SWE-bench to models tested without thinking isn't a model comparison. It's a methodology comparison.
Some models are tested with thinking off. Kimi K2.5 was benchmarked in instant mode (thinking explicitly disabled) because their eval harness was incompatible. Their 76.8% SWE-bench is their floor, not their ceiling.
Thinking can hurt. DeepSeek V3.2 is worse at coding with thinking on: 50.7% vs 56.1% on LiveCodeBench. But thinking helps its math. DeepSeek switches modes between benchmarks.
Handicapped
Kimi K2.5, DeepSeek V3.2
Unfair advantage
MiniMax M2.5 (always on)
Thinking hurts code
DeepSeek V3.2 on LiveCodeBench
Full audit
Thinking mode status for every major model.
| Model | Thinking mode | What was published | Impact |
|---|---|---|---|
| Claude Opus 4.6 | Adaptive (always on) | Auto-adjusts reasoning depth. No off switch | Fair |
| MiniMax M2.5 | Always on (architectural) | Interleaved thinking baked in. Cannot be disabled | Unfair advantage vs models tested without thinking |
| Kimi K2.5 | Toggle (instant vs thinking) | Benchmarked in INSTANT mode: thinking OFF for SWE-bench, AIME, GPQA | Handicapped. Real scores unknown |
| GLM-5 | Always on | Reasoning integrated into architecture | Fair |
| DeepSeek V3.2 | Toggle | Thinking HURTS coding: 50.7% vs 56.1% non-thinking on LCB | Mode-switching between benchmarks |
| Gemma 4 26B / 31B | Toggle (think: true) | Thinking ON for all published benchmarks | Fair |
| Qwen 3.5 | Toggle | Thinking ON for published benchmarks | Fair |
Highlighted rows are the problematic ones. The thinking mode status changes the interpretation of every published score.
SWE-bench annotated
The coding leaderboard nobody publishes.
Same scores every leaderboard shows. Different column: whether thinking was on.
| Model | SWE-bench | Thinking? | What this means |
|---|---|---|---|
| MiniMax M2.5 | 80.2% | Always ON | Fair score, but unfair comparison |
| GLM-5 | 77.8% | Always ON | Fair score |
| Kimi K2.5 | 76.8% | OFF (instant) | 3.4% gap with MiniMax might not exist |
| Qwen 3.5 397B | 76.4% | ON | Fair score |
| Qwen3-Coder-Next | 70.6% | ON | 80B MoE, 3B active |
| DeepSeek V3.2 | 67.8% | OFF (for code) | Thinking hurts its coding. Non-thinking intentional |
If Kimi K2.5's thinking-on SWE-bench lands above 80%, the entire leaderboard reshuffles.
Their eval harness couldn't handle thinking mode output at publication time.The Kimi question
What if Kimi K2.5 had thinking on?
Kimi scores 87.6% on GPQA with thinking off. MiniMax scores 47% with thinking always on. That's a 40-point gap in the wrong direction: the handicapped model crushes the unhandicapped one on science.
On math, Kimi hits 96.1% on AIME with thinking off: the highest score of any model, period. MiniMax estimates sit at 78-88%.
The implication: Kimi K2.5 with thinking on might be the best all-around model in the open-weight world. We just don't have the numbers.
| Benchmark | Kimi (OFF) | MiniMax (ON) | Kimi (ON) |
|---|---|---|---|
| SWE-bench | 76.8% | 80.2% | Unknown |
| AIME 2026 | 96.1% | ~78-88% | Unknown |
| GPQA Diamond | 87.6% | 47% | Unknown |
| LiveCodeBench | 85.0% | Unknown | Unknown |
GPQA Diamond -- Kimi (off) vs MiniMax (on)
Kimi K2.5 (thinking OFF)
MiniMax M2.5 (thinking ON)
The handicapped model crushes the unhandicapped one by 40 points on science. The leaderboard ranking depends entirely on which benchmark you look at.
The DeepSeek paradox
When thinking makes code worse.
DeepSeek publishes coding scores with thinking off and math scores with thinking on, switching modes between benchmarks.
| Benchmark | Non-thinking | Thinking | Difference |
|---|---|---|---|
| LiveCodeBench v6 | 56.1% | 50.7% | -5.4 pts |
| AIME 2026 (math) | 83.5% | 86.3% | +2.8 pts |
| Implication | Published coding scores | Published math scores | Different mode per benchmark |
What to do about it
How to read benchmarks honestly.
Rule 1
Always check the thinking mode status. Model cards and paper appendices sometimes disclose this. If they don't, assume the benchmark used whatever mode produces the highest score.
Rule 2
Compare models at the same thinking mode. MiniMax M2.5 (always on) vs Kimi K2.5 (thinking off) isn't a fair fight. Wait for normalized scores, or discount the always-on model's advantage.
Rule 3
The scaffold matters more than the model. SWE-bench Pro shows a 22+ point swing between different agent scaffolds using the same model. That's 3x the gap between the best and worst models on this leaderboard.
The bigger picture
Context is the multiplier, not the model.
The model is one variable. The retrieval infrastructure (how context is retrieved, how tasks are decomposed, how code is navigated) matters more than which model sits behind it.
A 22-point scaffold swing on SWE-bench Pro vs a 3-point model gap. The thinking mode debate matters for benchmark integrity, but in production, the context window is the bottleneck, not the reasoning chain.
That's where Interlinked Files lives. Machine-wide retrieval that turns any model into a better coding agent, regardless of whether thinking is on or off.