Verify what minds can actually do — silicon and carbon, under one instrument.
A local-first scientific instrument that runs blind capability tests on AI models — and on you — with machine-verified ground truth and evidence you can audit. No account. No cloud requirement. Your hardware, your data, your verdicts.
macOS, one line — installs the instrument and PostgreSQL, wires up a service that survives reboots, and opens the dashboard:
curl -fsSL https://calibrationscope.com/install.sh | sh
Prefer plain Homebrew? Same self-contained binary; you point DATABASE_URL at your own Postgres:
brew install it-help-san-diego/tap/calibration-scope
Local-model runs need LM Studio serving on :1234; cloud models just need your own API keys, entered on the setup page. Linux prebuilts, build-from-source, and full docs: github.com/IT-Help-San-Diego/calibration-scope.
Hardware reality, disclosed: the instrument itself is featherweight — it runs on nearly anything, ARM boxes included. Hosting models is the heavy part, and let's be honest about the iron: a small machine tops out around the 2B-parameter floor our own leaderboard measured; "local AI at frontier quality" means something like a maxed-out MacBook Pro — a several-thousand-dollar supercomputer in a laptop costume; and "cloud" just means renting someone else's data center by the token. On weak hardware you're benchmarking cloud models — and yourself. The scope runs on a $60 board; the minds it measures do not. Honest instrument, honest hardware bill.
The binary is self-contained — the dashboard and the SHA3-pinned test stimuli travel inside it, and every release ships with GitHub build attestations you can verify. To audit your evidence database, any Postgres client works; we like TablePlus. The schema is the API. Read install.sh before you pipe it — we would.
A human writes a sentence. The sentence becomes a theorem. The theorem is machine-checked against the model's answer — not a vibe, not a benchmark score someone else published. This is how humans tell computers what must be true, and then hold every mind — silicon and carbon — to it.
Every test in the battery carries its formal spec beside the prompt. When a model fails, we check the instrument first — the same rule applied to the human calibration arm. Theorems are introduced, not asserted.
Every week a new model claims to reason. The claims arrive wrapped in benchmarks the model may have memorized, scored by graders nobody audits, summarized in press releases nobody can reproduce. That isn't measurement — it's marketing with numbers on it.
Calibration Scope takes the opposite bet: a verdict is only worth what its evidence chain can prove. Every test has formal ground truth checked by a complete decision procedure — truth tables and finite model search, 2,400 years of logic, machine-checked. Every run is sealed with SHA-3 the moment it completes. Every stimulus is pinned by hash so nobody — including us — can quietly swap the question after seeing the answer.
A real measurement beats a confident reading. That is the Verification Principle.
Formal logic under paraphrase and adversarial traps — does the verdict track the structure, or the wording? Modus ponens to quantifier swaps, every answer machine-verified.
Ground-truth visual questions with pinned images. Reads what's there — or hallucinates what it expects? We caught frontier models doing the second. It's in the data.
Structured tool calls that must parse, and injection attacks that must be refused. Pass both or the machine isn't ready for real work.
The same battery, taken by you. Signal score: did you get the logic right? Carrier variance: did the wording change your verdict? Carbon and silicon, same scale.
✓ Every verdict sealed: SHA-3 provenance · pinned stimuli · reproducible on your machine
Running this instrument against dozens of local and frontier models produced findings we didn't expect — and some we did:
The carrier changes the verdict. Give a small model identical logic wrapped in different prose, and the verdict tracks the wrapper, not the content. Formal notation — the "most rigorous" encoding — turned out to be the heaviest noise. Poetry was the gentlest. Big models shrug all of it off; small ones don't. The wrapping paper matters until the mind is strong enough that it doesn't.
Local models can match the frontier. On sealed reasoning batteries, the best models that run offline on consumer hardware match the flagship cloud models — at zero dollars per query. The gap is real at the bottom and gone at the top.
The instrument fails more often than the subject. A wrong Unicode comparison, a token budget too small, a grader that punished honest refusals — most "model failures" we chased turned out to be our failures. So we started keeping receipts on ourselves too. That's what the Lessons page is: the failure runs of the humans who built this, documented with the same discipline we demand of the machines.
The instrument is a single binary with a PostgreSQL database, running entirely on your machine at 127.0.0.1:8768 (direct, loopback-only — local.calibrationscope.com resolves there too on machines with the DNS record). It drives your local models directly, keeps every trial in a database you own and can open, and never phones home. This website is documentation and findings — the science happens on your hardware, where you can check it.