Docs

Install

The package is tiltmeter on npm, run through npx so there is nothing to install globally:

npx tiltmeter@1 --help

init, lint, plan --offline, and verify all work with no ANTHROPIC_API_KEY and no network — proven against the packed npm tarball installed fresh into an empty directory, not just inside this monorepo.

Five-minute quickstart

No API key for any of this. It scaffolds a suite from a real directory of skill descriptions, checks it, and produces a cost estimate for running it — the exact trio a stranger can run cold.

1. Point init at a directory of <skill-name>/SKILL.md files (the Claude Code skills convention — any directory with that shape works, including a small test one you write by hand):

   npx tiltmeter@1 init --from-skills ./skills

   Writes a suite file under observatory/suites/ with one TODO-scenario item per skill description found, plus a panel and pricing manifest if you don't already have one.

2. Check the scaffold — schema, the negatives quota, provenance, token headroom:

   npx tiltmeter@1 lint

   Fails loudly on a suite with too few negatives or an item missing its source provenance — both are deliberate, not warnings.

3. Build the run matrix and get an exact cost estimate without spending anything:

   npx tiltmeter@1 plan --run-group demo-1 --offline

   --offline falls back to a heuristic multiplier instead of calling the (free) count_tokens endpoint, and marks the resulting estimate approximate so it is never confused with a real one.

Replacing the TODO scenarios init wrote with real ones, then running tiltmeter run --plan demo-1, is the only step in this whole quickstart that spends money — and it is a separate, explicit command, never something plan or lint does on your behalf.

The attribution model

Every reading carries an axis tuple — five hashes identifying which suite, which model, which runner behavior, which presentation template, and which sampling policy produced it. Two readings are compared only when exactly one of those five differs. Model differing (same run group) reads as the tilt this project is named for; nothing else differing, across run groups, reads as the published noise floor; a suite hash differing (same run group) reads as a rebaseline, your own harness edit checked against itself.

Anything else — most commonly a harness edit landing in the same window as a model release — is cannot-attribute, published with the exact axis names that co-varied. It is never guessed and never silently dropped; the chart draws a labelled break instead of a line.

The statistics

Each item has a pass fraction (passes out of k repeats). A suite's delta is the mean pass-fraction difference between two readings, computed with a seeded paired percentile bootstrap over items — items, not trials, resampled together on both sides so that item-difficulty variance cancels out. The seed is the first eight hex characters of a hash of both readings' body hashes, so a rerun of the same comparison always draws the same resamples; nobody picks it by hand. A verdict of regressed or improved requires both a 95% confidence interval that excludes zero and a delta at least as large as the minimum detectable effect (one item's worth, by default). Anything smaller is moved-within-noise.

Sampling runs at k=3 repeats, temperature 1.0 — not temperature 0. Production harnesses run at default sampling, so temperature 0 measures a configuration nobody ships. Temperature 0 is also not deterministic on provider infrastructure, so it would buy false confidence rather than real reproducibility. And k repeats at temperature 1.0 produce a per-item pass fraction — a flakiness signal that temperature 0 hides entirely, since it always resamples the same single trajectory. There is no seed parameter on the Messages API; that irreducible non-determinism is why every run group also carries a null pair (below).

Item immutability

A suite's items cannot be edited in place. tiltmeter lint compares each item's canonical bytes against the version last published in a reading, or against the previous commit that touched the suite file when no reading references it yet, and fails on any in-place change. Changing an item means retiring it — it stays in the file, visible on the site — and adding a new one with a new id.

This is deliberate friction: it is the anti-p-hacking mechanism. Nobody can quietly delete the item a new model failed and call the suite unchanged, because the check runs from git history alone in CI, not from anyone's memory of what the suite used to contain.

Cost model

Two independent cap layers. A provider-enforced monthly spend limit, set on a dedicated Anthropic API key in its own console workspace — the only cap that still holds if the key itself leaks, because it is enforced by Anthropic's infrastructure rather than this repo's code. And a runner-enforced set of caps tracked from the committed readings index: maxRunUsd = $3.00, maxCellUsd = $1.50, maxMonthUsd = $15.00.

plan refuses to emit a plan that would exceed any cap. run re-checks against actual spend after every cell and, on a breach, stops submitting, writes the reading as aborted, and commits that fact — never a silent skip. Batch API submission, at half the synchronous rate, is the default mode.

The secret boundary

ANTHROPIC_API_KEY is read from the environment only — never a CLI flag (shell history would keep it), never written to disk, never sent anywhere but api.anthropic.com. The site itself has no key field, no server, and no request path that could reach a model — it is a static export with zero API routes.

The two scheduled workflows that use the key run on a schedule or a manual dispatch only, never on a pull request or any other externally-triggerable event — checked mechanically in CI (scripts/lint-workflow-secrets.mjs), not just by review, so a workflow file that starts reading the key from a fork-reachable trigger fails the build.

Failure modes

Limitations

Every limitation that bounds what a number here is allowed to mean is stated in full, in one place, on methodology. It is not repeated here: a list kept in two places is a list that drifts, and this one already had.