Guides

Contract reports

Public package status: the Python distributions and desktop installers are not released yet. These pages document an evolving interface, not currently usable install artifacts. View release status.

The benchmark contract layer can tell you three things about itself: how every shipped bundle scores against the per-bundle checks (the fleet report), what every registered rule asserts and how to fix a violation (the rule book), and what an eval submission YAML may declare (the submission reference). None of these is a committed document. Each one is rendered on demand, at the moment you ask, from the same code that enforces it.

That is a deliberate design (2026-08-22): a committed report is a cache of a command’s output, and a committed rule book is a transcription of the registry — both go stale the moment the code moves, and a stale document that reads like a spec is worse than no document. The single source of truth is medotter_core/contracts/spec.py (the field registry), the check functions themselves, and each bundle’s own bench.yaml.

All three commands below run from a checkout of medotter/medotter, at the repository root.

The fleet report

python3 -m medotter_core.contracts report --all --format md

Runs every per-bundle check against every shipped bundle and prints one markdown section per bundle (plus the corpus-wide dataset pass), each with a per-check verdict table and a FAIL/VACUOUS/SKIP/PASS tally. The handful of suite-level checks that judge the fleet as a whole run under report --suite instead. For a single bundle, name it instead of --all:

python3 -m medotter_core.contracts report medotter/benchmarks/glioma_wt_tc_et --format md

You do not need to run this to see the current fleet state: each run of the SDK tests · CPU + wheel workflow that reaches its summary step publishes the same report into the run’s job summary on GitHub Actions, so the freshest copy is one click away from the latest commit’s checks.

The rule book

python3 -m medotter_core.contracts rules

Renders every registered rule as markdown: its subject, lane, the assertion quoted from the check’s own docstring, the remedy derived from the field registry, and a constructed counter-example for every closed alphabet. Every value in the output is interpolated from the registry at call time — the renderer holds no copies, and a test suite pins that property.

For a quick registration-level summary of one rule (its contract, lane, field and inputs — not the full rule-book section):

python3 -m medotter_core.contracts explain BENCH-LABEL-03

The submission-YAML reference

python3 -m medotter_core.contracts submission-rules

Prints the field table and enforcement map for the submission YAML that python -m medotter evaluate eval.yaml (or its contributor-side door, python -m medotter.agent evaluate eval.yaml) consumes: every key, its type, when it is required, what defaults apply on each model regime, and where each rule is enforced — the load-time gate that refuses a violation, and the audit check that re-judges the same predicate on a finished run record. The text lives beside the spec that enforces it (medotter_core/contracts/eval/submission_spec.py), and its footer — schema id, key whitelists, resolver defaults — is read off that spec at call time. The eval entry point’s own error messages point at this flag.