Leaderboard

Bundle schema

Public package status: the Python distributions and desktop installers are not released yet. These pages document an evolving interface, not currently usable install artifacts. View release status.

A bundle is a frozen benchmark task: a directory of files that is the single source of truth for which cases exist, how they’re split, and how predictions on them get scored. medotter_core.bundle.Bundle is the loader both the website and the eval worker use to read one — its docstring describes a bundle directory as containing bench.yaml, manifest.json, bench.resolved.yaml, and coverage_report.{json,md}. Of those, Bundle itself only reads manifest.json eagerly (at construction) and bench.yaml / bench.resolved.yaml lazily (only when you call .task_spec() / .eval_spec()); coverage_report.* is documentary — human/website-facing coverage notes, never read by Bundle’s own API.

Those four files don’t all live in one place. The medotter SDK repo’s benchmarks/<name>/ directories are the factory side: all nine bundles there carry a single declaration face, bench.yaml (plus its frozen bench.final.yaml snapshot), and git-track their built manifest.json — no bench.resolved.yaml exists in that repo; the second eval-face carrier is a web-side materialisation only. This website repo (MedOtter-Web) is the consumer side: it vendors full, frozen copies of each bundle under evaluation/bundles/<name>/, and the vendored copy may additionally include coverage_report.{json,md} — currently only evaluation/bundles/glioma_wt_tc_et/ ships them, alongside bench.yaml, bench.resolved.yaml, and manifest.json (the other vendored bundles carry just the first three). That vendored copy is what the eval worker actually loads at runtime.

This page documents the real field names, read directly from medotter_core/bundle.py and the bundle directories under medotter/benchmarks/ in the SDK repo — not invented.

Bundle

from medotter_core.bundle import Bundle

# thoracic_lesion_ct: like every bundle in the medotter repo, its
# manifest.json (and bench.yaml, and bench.final.yaml) is git-tracked there,
# so Bundle() — which eagerly reads manifest.json at construction — works on
# any of the nine straight from a checkout.
b = Bundle("benchmarks/thoracic_lesion_ct")
b.task      # manifest["task"]     -> "thoracic_lesion_ct"
b.version   # manifest["version"]
b.hash      # manifest["hash"]
b.train     # manifest["train"]    -> list of case ids
b.test      # manifest["test"]     -> list of case ids
b.case_meta("COVID_19_20/volume-covid19-A-0003")   # manifest["per_case"][case_id]
list(b.cases())                          # (case_id, meta) for every case, train+test
b.task_spec()   # yaml.safe_load(bench.yaml)   — lazy, re-parsed each call
b.eval_spec()    # yaml.safe_load(bench.resolved.yaml) — lazy, re-parsed each call

bench.yaml

The task ontology: what the benchmark measures, its canonical label space, and (the real, populated remap mechanism) the per-dataset native→canonical label id map. Real fields, grounded in benchmarks/thoracic_lesion_ct/bench.yaml and benchmarks/abdominal_multiorgan_ct/bench.yaml:

# benchmarks/thoracic_lesion_ct/bench.yaml — trimmed
task:
  id: thoracic_lesion_ct
  name: Unified Thoracic Lesion Segmentation (CT)
  modality: CT
  anatomy: thorax / lung
canonical_labels:
  tumor_nodule: [1]
  infection:    [2]
main_region: macro_avg
label_remap:
  NSCLC_Radiomics_Interobserver1: {1: 1}
  MIDRC_RICORD_1a: {1: 2, 2: 2, 3: 2}   # native 4/5/6, not listed, drop to bg

manifest.json

The frozen per-case file listing and train/test split. Required keys, per Bundle’s accessors: task, version, hash, train, test, per_case. Real manifests in this repo also carry split_version, and the newer ones add skipped / dropped_cross_split bookkeeping arrays — those aren’t read by Bundle, but they’re part of the real on-disk shape.

{
  "task": "vessel_ct",
  "version": "0.1.0",
  "split_version": "vessel_ct/v1",
  "hash": "facbb5f45b72",
  "train": ["FLARE22/FLARE22_Tr_0001", "..."],
  "test": ["BTCV/0001", "..."],
  "per_case": {
    "BTCV/0001": {
      "dataset": "BTCV",
      "split": "test",
      "tier": "test",
      "patient_uid": "0001",
      "image": "btcv/RawData/Training/img/img0001.nii.gz",
      "seg": "btcv/RawData/Training/label/label0001.nii.gz"
    }
  }
}

per_case[case_id] fields, observed across bundles in this repo (the exact set varies by bundle/generation):

All manifest-supplied paths are resolved relative to data_root (or its _prepared/ subdirectory when present) and validated against absolute paths and .. traversal before being opened.

bench.resolved.yaml

The scoring protocol: which regions/classes are scored, which metrics, and (for bundles that declare it) which adapter runs the scoring. Real fields, observed across bundles in this repo:

Two real examples, trimmed:

# benchmarks/thoracic_lesion_ct/bench.resolved.yaml — semantic seg, task_type
# absent (implicitly "seg")
mode: 3d_volumetric
segmentation_mode: multi_class
eval_regions:
  tumor_nodule: [1]
  infection: [2]
partial_label: true
main_region: macro_avg
metrics: [dice, hd95]
# benchmarks/nucleus_instance_2d/bench.resolved.yaml — instance seg, task_type
# explicit
task_type: instance_seg_2d
mode: 2d_image
segmentation_mode: instance
eval_regions:
  nucleus: [1]
partial_label: false
main_region: nucleus
metrics: [f1, aji, pq]
iou_threshold: 0.5

Task types

Registered in medotter_core/adapters/registry.py, populated by medotter_core/adapters/__init__.py. Implemented (a submission can actually be scored against these):

task_typeAdapterWhat it scores
seg (default)SegAdapterPrompt-free 3D volumetric semantic segmentation — the current leaderboard path (glioma, abdominal, liver, vessel, thoracic bundles).
seg2dSeg2DAdapterPrompt-free 2D per-image semantic segmentation (histopathology / microscopy tiles).
instance_seg_2dInstanceSeg2DAdapter2D per-image nucleus instance segmentation — F1@IoU / mAJI / PQ, IoU-matched ids.
instance_seg_3dInstanceSeg3DAdapter3D per-volume nucleus instance segmentation — same instance metrics over a whole volume.
prompt_seg(SAM/SAM2-style adapter)GT-prompted segmentation. Not eligible for the untrusted leaderboard worker: its inference derives prompts from ground truth, so it cannot run GT-free in the worker’s sandboxed child — the adapter’s infer_iter refuses, and evaluation only runs one-process (local/offline).

Also registered, but not implemented — stub adapters that raise NotImplementedError if a bundle declares them: cls, det, vlm.

Next: this closes the submitter track. See Contract for what your model needs to expose, and Guides → Evaluation for the SDK-side scoring API these bundles share metrics with.