tiltmetertiltmeter

tiltmeter

pre-release · 2026-08-09

The detector fires when your harness moves, and not when it doesn't.

A model release either changed how your skill descriptions, tool schemas and output contracts behave, or it didn't. Both answers are only worth having from an instrument whose error rates are known — so those were measured first, before a single reading was taken.

400 seeded trialsB = 10,000 resamples each40-item pool8 plantedpnpm calibration$0
Calibrationboth gates cleared
False positives
0.0%0/200 fired
ceiling 5.0%
Detection power
95.0%190/200 caught
floor 90.0%

Both calibration gates, cleared

methodology →

A seeded simulation, re-run by CI on every push and failed on drift — never a threshold picked by eye. 4,000,000 bootstrap resamples in total.

False positives

n=200

0.0%0/200 fired

ceiling 5.0%5.0 pts under ceiling

one cell per trial — every one quiet is a trial that did not fire

Detection power

n=200

95.0%190/200 caught

floor 90.0%5.0 pts over floor

one cell per trial — 10 misses shown quiet

The schedule has already run 4 times

chain verified unbroken at build

Every run committed a record and linked it to the one before it. All 4 refused, for the same stated reason, and said so in public rather than publishing a number nobody paid for.

  1. rg-20260810-12026-08-10skipped

    ANTHROPIC_API_KEY not set · planned $0.7984, spent $0.00

    genesis 0e74605383

  2. rg-20260817-12026-08-17skipped

    ANTHROPIC_API_KEY not set · planned $0.7984, spent $0.00

    0e74605383 → 228ff518d9

  3. rg-20260824-12026-08-24skipped

    ANTHROPIC_API_KEY not set · planned $0.7984, spent $0.00

    228ff518d9 → a96569107e

  4. rg-20260831-12026-08-31skipped

    ANTHROPIC_API_KEY not set · planned $0.9980, spent $0.00

    a96569107e → d45759f34c

What it caught while waiting

The identical plan — 12 cells, 324 item runs, unchanged — repriced itself when a standing model's published price moved.

$0.7984
rg-20260824-1
$0.9980
rg-20260831-1

+25.0% for the same work, on sonnet5 — read from the dated pricing manifest, not from a hardcoded rate.

caps $1.50/cell · $3.00/run · $15.00/month
pricing manifest pricing.2026-08-08

How a comparison resolves

A cell's identity is five hashes: suite, model, runner behavior, presentation, sampling policy. Change exactly one and the comparison resolves to a verdict. Change two and it refuses, naming the axes rather than guessing.

tiltmeter's axis tuple: one difference is a verdict, two is a refusalTwo stacked comparisons of the same five-element axis tuple (suite, model, runner, presentation, sampling). Top: reading A and reading B differ on exactly one axis, model — the comparison resolves to a verdict (regressed, improved, or moved-within-noise). Bottom: reading A and reading B differ on two axes, suite and model at once — the comparison refuses to resolve and is published as cannot-attribute, drawn in amber because refusing to answer is the argument this project makes.ABverdictregressed / improved /moved-within-noisesuitemodelrunnerpresentsamplingone axis differs (model) → attributableABcannot-attributereasons: suiteSpecHash +modelIdResolvedsuitemodelrunnerpresentsamplingtwo axes differ (suite + model) → refused, not guessed
Two readings compared on the five-element axis tuple (suite, model, runner, presentation, sampling). Change exactly one and the comparison resolves to a verdict. Change two — a suite edit landing in the same window as a model release, the one case an operator actually has to untangle — and the comparison refuses to resolve: cannot-attribute, published with the axes named, never guessed.

See it run

Recorded against this site as deployed, not a local build — if the site were broken, the recording would be too.

The tiltmeter landing page as deployed, showing the calibration readout — the first frame of the recording.

Motion is reduced on this device, so the recording does not autoplay. Play the recording (webm, muted, no audio).

A scripted run against the real deployed site (not a screen capture): the landing page, then /models — proving no leaderboard exists there — then /methodology.

The four launch suites

108 items · 34 negative (31.5%)

Committed on 2026-08-09, before any reading and before whatever release eventually moves one of them exists. A suite that only ever tests the happy path cannot tell you when it starts firing on everything — so every suite carries a negatives quota.

house-skill-activation

32
active items
12
negative
37.5% negativequota met

suiteSpecHash 6cbcf2e

mcp-tool-selection

28
active items
8
negative
28.6% negativequota met

suiteSpecHash 9981576

output-contract

26
active items
8
negative
30.8% negativequota met

suiteSpecHash 39fd16e

routing-adherence

22
active items
6
negative
27.3% negativequota met

suiteSpecHash e5e7d16

How pre-registration is checked, not claimed

For each reading, verify recomputes suiteSpecHash from the suite file, walks git history for the first commit whose tree contains that hash, reads the model's cited release date, and asserts the suite was registered first — printing the commit SHA and both dates. Git history is the actual proof; the command just makes checking it a 30-second job.

npx tiltmeter@1 verify

tiltmeter launched 2026-08-09 with 4 pre-registered suites and 108 items. There is no time series yet — that is what pre-registration means. The series starts here.

The first run group spends real money and is a deliberate, gated step. Until it is taken, the schedule keeps publishing what it did instead — 4 records so far, priced against a pricing manifest fetched 2026-08-08.

Install

The first three commands run with no API key and no network — proven against the packed npm tarball in a clean directory, not just in this repo.

npx tiltmeter@1 init --from-skills <dir>
npx tiltmeter@1 lint
npx tiltmeter@1 plan --run-group <id> --offline

Full walkthrough on docs.