Verify a benchmark or paper
Independently check that a repo's reported numbers reproduce — and aren't contaminated.
The core job: point Calma at a benchmark, eval, or paper repository and get an independent, third-party verdict on whether its reported numbers actually hold up. No self-grading — a deterministic re-run and recompute, neutral by construction.
This is the workflow for benchmark maintainers, eval vendors, artifact-evaluation committees, and anyone who needs to trust a number before they publish or buy on it.
What Calma checks
| Question | How | Verdict |
|---|---|---|
| Does the reported metric reproduce? | Re-run the repo, recompute the metric from the raw outputs, diff it | CONFIRMED / REFUTED |
| Do the committed result files regenerate? | Re-run the repro script, diff regenerated vs committed cells | reproduced / not-reproduced |
| Is the benchmark contaminated? | Train/test overlap (verbatim + near-duplicate) detection | INVALIDATED on leakage |
Run it
# recompute the reported metric + check the committed result files regenerate
calma verify owner/benchmark-repo --deep --artifact-repro
--deepclones into a network-isolated sandbox, re-runs the repo, and recomputes each reported number.--artifact-reproadditionally diffs the repo's committed result files (leaderboard CSVs,results.json, per-run metrics) against what a fresh run regenerates — cell by cell. A number that doesn't come back is flaggednot-reproducedwith the exact file and cell.
If the repo's run command needs a config or data argument, name it:
calma verify owner/repo --deep --artifact-repro --entry "reproduce.py --config configs/main.yaml"
Two verifiable shapes
Calma lands a clean verdict when a repo has a runnable entrypoint plus one of:
- Committed predictions + labels + a reported score → Calma recomputes the standard metric and returns CONFIRMED / REFUTED. (Live leaderboards usually hide labels; reproducibility artifacts commit both.)
- A repro script + committed result files → artifact-reproduction diffs every regenerated cell against the committed one.
Why the verdict is trustworthy
The verdict is a single deterministic function of the recompute — it is fail-closed and can never be a false confirm. When Calma can't bind a number to a computation, it says INCONCLUSIVE rather than guessing. That is the whole point of a neutral auditor: a CONFIRMED means the number reproduced, full stop.
Gate it
calma verify owner/repo --deep # exits non-zero on REFUTED / INVALIDATED
Drop that into CI (or the GitHub Action / PR check) so a regression that breaks a reported number fails the build.