Opus 4.7 · five effort levels · quality
Same model, different effort. How much better does it actually get?
Claude Opus 4.7 has five reasoning settings: low, medium, high, xhigh, max. We ran the same task five times, once at each level, on the same five real TypeScript files from the OpenClaw CLI, and read every output line by line.
For context, we also pulled four baseline models from the earlier OpenClaw benchmark: Gemma 26B Q8 Think (local, RTX 5090), Claude Haiku 4.5, Claude Sonnet 4.5, and Claude Opus 4.6. Same five files. Same output schema. Apples-to-apples.
The headline is nuanced: five of nine models sit in a 0.3-point peer band (8.3 to 8.6), tied within noise. Only Opus 4.7 high / xhigh / max clearly step above. Only Haiku clearly sits below, because it hallucinated on a types file. Every quality number here was assigned after spot-checking specific claims against the 755-line source code line by line.
The headline · graded on 10-point scale
Opus 4.7 max: 9.5/10 a full point above prior frontier
Scores use the same 10-point scale as the prior OpenClaw benchmark. Each one was assigned after reading every output and spot-checking the specific claims against the 755-line source file line by line. Of 45 output JSONs across 9 models, only one produced a claim the code doesn’t support: Haiku’s speculation about resolvability semantics on a types-only file.
Five runs
Every effort level, one table.
Quality is a single 0 to 10 score per model, assigned after reading every output and verifying the specific claims line-by-line against the 5 source files. Same scale as the earlier OpenClaw benchmark. Click any column to sort.
| Model / effort | Wall time (5 files) | Quality (0 to 10) ↓ | Hallucination? | Verified findings |
|---|---|---|---|---|
| Opus 4.7 · max | 282s | 9.5 | no | Only model to catch WEB_RUNTIME_SECRET_TARGET_ID_PREFIXES constant + preflight gateway-skip + fail-open design |
| Opus 4.7 · xhigh | 529s | 9.2 | no | Caught sudo stdio pattern + commitLabel fallback + exact regex; slight discipline slip on types file |
| Opus 4.7 · high | 104s | 9.0 | no | First to enumerate all 4 exact unsupported-method substrings; explains WHY setPathExistingStrict prevents schema drift |
| Opus 4.7 · low | 103s | 8.6 | no | Named structuredClone + exact marker string + provider-match rule: things 4.6 / Sonnet / Haiku missed |
| Opus 4.7 · medium | 95s | 8.6 | no | Indistinguishable from low on this test: same claims, same discipline, same 0 hallucinations |
| Claude Opus 4.6 | ~51s | 8.5 | no | Legacy alias mappings explicit; exact inspect format; slight mega-tag inflation on types file |
| Gemma 26B Q8 Think | ~33s | 8.4 | no | Most connects_to (14) on hardest file; compressed legacy aliases correctly; perfect discipline on trivials |
| Claude Sonnet 4.5 | ~48s | 8.3 | no | Perfect discipline; 30s timeout named; every claim verified correct but less depth than 4.6 on hardest file |
| Claude Haiku 4.5 | ~22s | 7.2 | yes | Invented schema-validation tag on types file; speculated about resolvability semantics with no code evidence |
Five models sit in a 0.3-point peer band (Sonnet 8.3, Gemma 8.4, Opus 4.6 8.5, Opus 4.7 low & medium 8.6). On a 5-file sample with its noise, these are statistically tied with different shapes: Gemma wins discipline + entity coverage, 4.6 wins QA depth, 4.7 low wins mechanism naming. Only Haiku sits below the band because the hallucination is a real quality problem, not a cosmetic one. Opus 4.7 high / xhigh / max step above the band by catching invariants every peer-tier model missed.
Two time anomalies worth flagging: xhigh (529s) took almost twice as long as max (282s) on the same 5 files. Probably session variance: xhigh thought hard on every file including trivial ones; max allocated budget more intelligently. And Gemma / Haiku / Sonnet / Opus 4.6 wall times are approximate (pure API round-trips), while Opus 4.7 times include Claude Code tool overhead. Compare trends within columns, not pure wall-clock across plumbing.
Peer tier (5 models)
8.3 to 8.6. Different shapes, same tier.
Sonnet, Gemma, Opus 4.6, Opus 4.7 low & medium all land inside a 0.3-point band. Different strengths: Gemma wins discipline and cross-ref count, 4.6 wins QA depth with explicit legacy-alias mappings, 4.7 low wins mechanism naming (named structuredClone, exact marker strings). On 5 files, distinguishing within this band overclaims.
High is the real step up
9.0. Same wall time as low.
High is where depth actually climbs (8.6 → 9.0) and comes at essentially the same wall time as low and medium. First to enumerate all 4 exact error-string substrings for unsupported-method detection, verified line-by-line against the source. If you only pick one effort level as a default, this is the one.
Max catches what nobody else did
9.5. 1 full point above 4.6.
The only level to spot the preflight gateway-skip, name the WEB_RUNTIME_SECRET_TARGET_ID_PREFIXES constant, and call out the fail-open-on-unknown-mode design. For canonical documentation of small corpora where every invariant matters, max is the one a human reader would trust.
Quality (0-10 scale) · 9 runs
Eight out of nine cluster tight. One sits below.
Bar length = quality score out of 10. Most models land inside a 1.2-point band (8.3 to 9.5): they’re all correct, all specific, differing in depth not accuracy. Haiku sits a point lower because it hallucinated on the types file. Wall time at right (approximate for API models, actual for Opus 4.7).
The tight peer band (8.3 to 8.6) says something real: on a 5-file sample, calling any of these 5 models “better” than another inside the band is noise. Opus 4.7 high is the first model that visibly steps out (+0.4 above the band at the same wall time as low). Max adds another +0.5, bought at ~2.7× the wall time. Haiku’s 7.2 isn’t about depth: it’s about accuracy.
Three hidden details from the OpenClaw files
What “quality” actually means here.
Three specific facts about the TypeScript code. Each is visible to any careful reader, but requires actually reading the function bodies rather than skimming the identifiers. The progression below shows which models caught which detail: a 3-to-2-to-1 ladder from “high and above” to “max only.”
4 exact error-string substrings for unsupported-method detection
isUnsupportedSecretsResolveError checks the lowercased error message for 'secrets.resolve' AND one of: 'does not support required method', 'unknown method', 'method not found', 'invalid request'. Every lower effort level said 'checks for unsupported method' generically. high/xhigh/max enumerated the exact four substrings.
sudo stdio=['inherit', 'pipe', 'inherit'] for password prompts
For the sudo-podman variant, stdio is deliberately split: stdin inherits so sudo can prompt for the password, stdout pipes so the `inspect` output is captured, stderr inherits so the prompt text reaches the user. Every other model said 'stdio inherit'; only xhigh and max noticed the three-element tuple and its purpose.
Preflight gateway-skip when no active or unknown refs
Before invoking the gateway RPC, the code classifies configured paths as active/inactive/unknown and early-returns if none are active or unknown, avoiding a pointless gateway roundtrip when all configured refs live on inactive surfaces. Only Opus 4.7 at max effort surfaced this optimization as a tag.
All nine models produce correct output on these files. What changes is how much they notice. Every prior frontier model caught zero of these three details. Every Opus 4.7 effort level caught at least one. The gap is not about accuracy: it’s about whether the model stopped at the surface or kept reading.
When to use what
Three defaults, no regret.
Scores 8.6, inside the peer band with Sonnet 8.3, Gemma 8.4, Opus 4.6 8.5. Medium ties low at 8.6: no reason to prefer it. Use low when you need fast structured output and plan to iterate later.
Scores 9.0 at the same wall time as low. The jump from 8.6 to 9.0 is the first step outside the peer band: high enumerates the 4 exact unsupported-method substrings, explains why setPathExistingStrict prevents schema drift, and names invariants the peer tier missed. Default dial for non-trivial work.
Scores 9.5, a full point above prior frontier (Opus 4.6 at 8.5). The only level that caught the preflight gateway-skip, named the WEB_RUNTIME_SECRET_TARGET_ID_PREFIXES constant, and surfaced the fail-open-on-unknown-mode design. Skip xhigh: max caught strictly more in half the wall time. Reserve for canonical documentation runs.
Related: original OpenClaw benchmark (9 models, 170 files) · Gemma vs API code analysis
Methodology
Every run analyzed the same five TypeScript files from the OpenClaw CLI, picked to span the complexity range: cron-cli.ts (1-line re-export), config-set-dryrun.ts (22-line types), banner.ts (160 lines), container-target.ts (302 lines), command-secret-gateway.ts (755 lines). Same system prompt, same output schema, identical inputs. The five Opus 4.7 runs used Claude Code at the stated /effort level; the four baseline runs (Sonnet 4.5, Opus 4.6, Haiku 4.5, Gemma 26B Q8 Think) came from the existing OpenClaw benchmark via harness.mts (direct API / Ollama).
Scores are a single 0 to 10 number per model, same scale as the prior OpenClaw benchmark (where Gemma scored 8.4 and Opus 4.6 scored 8.5). Assigned by reading every output, comparing it to the source code, and weighing four factors: discipline (did it return empty on trivial files or inflate?), accuracy (any claims the code doesn’t support?), specificity (does it name exact functions, constants, error strings?), and depth (does it catch non-obvious invariants a careful human would?).
Verification pass: every specific claim from all 45 output JSONs was spot-checked against the actual 5 source files: inspect format strings, regex patterns, stdio tuples, constant names, error substrings, line numbers. Nine of nine models produced accurate output on 44 of 45 files; Haiku produced one file (config-set-dryrun.ts) with claims the code doesn’t support. That one hallucination is the reason Haiku sits a point below the peer band.
Wall times are not perfectly apples-to-apples. Opus 4.7 runs include a constant ~2 to 5s Claude Code tool overhead per file; baseline runs are pure API. Differences within the Opus 4.7 column reflect actual reasoning time; cross-group comparisons should allow for the harness overhead. Also note the xhigh-vs-max time anomaly (529s vs 282s for the same files): probably session variance, a repeat run would smooth it.
A note on sample bias: 3 of 5 files here are non-trivial (160 to 755 lines), and the quality-grading weighs depth heavily. The full OpenClaw benchmark averaged over 170 files with many trivial ones, which boosts restraint-style models (Gemma, Sonnet). Both pictures are honest: they weight different strengths. Raw outputs: references/OpenClaw_Test/<model>/.