Methodology

Every scorer is a structural check over the response's tool_use blocks — never a read of assistant prose for meaning. A scorer that must interpret prose means the probe is wrong; the probe gets fixed, not a judge added.

repeats: k=3
temperature: 1.0
bootstrap B: 10,000
interval: 95% CI
min effect: 1/n

The eight scorers

deterministic only

Every scorer and the structural check it performs
Scorer	What it checks
tool-called(name, args?)	The first tool_use block matches a name, and optionally a subset of args.
no-tool-called	No tool_use block at all.
tool-in-set(names)	The first tool_use block's name is one of a declared set.
arg-enum(name, key, values)	The first call's argument at key is a member of a declared set.
arg-required-keys(name, keys)	The first call carries every required key, regardless of value.
tool-order(names)	The FULL sequence of tool_use blocks exactly matches a declared order.
literal-prefix(prefix)	The response's text starts with a literal, explicitly-demanded control token. Never used for prose meaning.
json-schema-valid(name, schema)	The first call's args structurally validate against a minimal JSON-Schema subset (type/required/properties/enum/items).

Presentation templates

4 committed

A presentation is a committed template — system-block layout, how each artifact kind renders into the request, tool_choice policy, stop conditions — hashed into every reading's axis tuple. Changing one invalidates comparison, deliberately.

output-contract@1

The ecosystem's own structured-output vocabulary, plus a decline channel.

system block676 chars

routing-policy@1

A routing decision surface: route / split_task / select_pattern.

system block1,198 chars

skill-tool@1

A skill-description artifact becomes one entry in a Skill tool's enum, mirroring Claude Code's own shape.

system block233 chars

tool-select@1

Real MCP tool-schema artifacts, rendered verbatim as tools[].

system block223 chars

What makes a number publishable

The axis rules

A cell's identity is its axis tuple: suiteSpecHash, modelIdResolved, runnerBehaviorVersion, presentationHash, samplingPolicyHash. A comparison is computed only when EXACTLY ONE differs. Anything else is cannot-attribute, published with every co-varying axis named — never guessed, never silently dropped.

The bootstrap and its bar

A seeded paired percentile bootstrap over ITEMS, not trials. The seed is the first 8 hex of sha256(bodyHashA + bodyHashB) — never chosen by the analyst. regressed/improved require BOTH the CI to exclude 0 AND the delta to meet the minimum detectable effect; otherwise moved-within-noise.

The noise floor

Every run group runs the cheapest panel model TWICE, as two distinct cells with identical axes — the null pair. The delta between them bounds what "moved" can mean that week, since nothing about the model or the harness differs. It is the negative control that makes every positive claim here credible.

Sampling: why not temperature 0

Production harnesses run at default sampling, so t=0 measures a configuration nobody ships — and t=0 is not even deterministic on provider infrastructure. k repeats at t=1.0 yield a per-item pass fraction, a real flakiness signal t=0 hides entirely.

Cost policy

per cell: $1.50
per run: $3.00
per month: $15.00

Two independent cap layers: a provider-enforced monthly spend limit on a dedicated API key — the only cap that survives a leaked key — and runner-enforced caps tracked from the committed readings index. plan refuses to emit an over-cap plan; run re-checks against ACTUAL usage after every cell and, on breach, stops submitting, writes the reading aborted, and commits — never a silent skip. Batch submission (−50%) is the standard mode.

Limitations

Stated in full, on the page, rather than in a footnote — every one of these bounds what any number here is allowed to mean.

The presentation here is not your production runtime — it approximates one shape, not every harness.
No seed parameter exists on the Anthropic API; exact reproduction of a single trial is not possible, only the reading's aggregate.
At k=3, no single item carries a confidence interval — inference lives at the suite level only.
Aliases can be substituted by the provider between run groups; when that happens it publishes as a labelled event, not a silent swap.
Anthropic-only panel at v1 — nothing here compares vendors.
Batch results may lag a model release by up to 24 hours.
These suites are James's own harness — they are evidence about this project's presentation, not a claim that they represent anyone else's harness, and never a claim about a model's capability in isolation.