Quickstart
Verify the numbers a repository reports — in one command.
Calma re-runs a repository, recomputes every number it reports from the raw outputs, and tells you which hold up — a deterministic verdict, never an LLM guess. It is the correctness layer for benchmarks, evals, and reproducible research.
There are three ways to use it, depending on where you work:
| Surface | Best for |
|---|---|
| Dashboard | Point-and-click: connect a repo, watch it verify |
| CLI | Your terminal or a CI gate — pip install calma |
| API / MCP | Automation, agents, and integrations |
60-second start (CLI)
pip install calma
export CALMA_API_KEY=calma_sk_... # create one at https://calma1.vercel.app/dashboard/keys
calma verify owner/repo --deep # re-run + recompute every number
That's it. Calma clones the repo into a network-isolated sandbox, re-runs it, recomputes each reported metric from the raw outputs, and prints a per-claim verdict.
What a verdict means
Every reported number gets exactly one verdict — the label is derived by a single deterministic function, so it is reproducible and can never be a false confirm:
| Verdict | Meaning |
|---|---|
| CONFIRMED | The number reproduces from a re-run + independent recompute |
| REFUTED | The code does not produce the reported number |
| INVALIDATED | The number reproduces, but the method is invalid (e.g. data leakage) |
| REPRODUCED_ONLY | Regenerates deterministically, but correctness couldn't be independently confirmed |
| INCONCLUSIVE | Ran, but the number couldn't be bound to a computation (fail-closed) |
| DISCOVERED | Found in the repo, not deep-verified |
Calma is fail-closed: when it cannot verify something, it says so — it never upgrades a doubt into a confirmation.
Next steps
- Authentication — create and use an API key
- CLI reference — every command and flag
- Re-run the paper — verify committed result files regenerate