3D volume benchmarks

load_benchmark gives you programmatic access to a benchmark bundle’s task metadata, and — for 3d_volumetric bundles — a way to run the same volume-level scoring the leaderboard’s evaluation worker runs.

This path is experimental. Benchmark.evaluate needs a prepared data_root (the bundle’s frozen manifest_test.json plus the actual case files, e.g. NIfTI volumes, on disk) that most bundles keep out of the development source tree — it lives on the evaluation worker / a DATA_ROOT box and is not a public artifact. The volume-eval path is also only partially proven on GPU today (it is exercised by a @slow @gpu tier5 test that skips, rather than fails, when DATA_ROOT or the manifest is absent on the runner). Expect the API here to evolve.

Listing and loading benchmarks

def list_benchmarks() -> list[str]
def load_benchmark(name_or_dir) -> Benchmark

import medotter as mo

mo.list_benchmarks()
# ['abdominal_multiorgan_ct', 'glioma_wt_tc_et', 'histopath_nucleus_seg',
#  'liver_lesion_ct', 'nucleus_instance_2d', 'nucleus_instance_3d',
#  'thoracic_lesion_ct']

bm = mo.load_benchmark("glioma_wt_tc_et")

name_or_dir is either a bundle name under the repo’s benchmarks/ directory, or a path to a bundle directory containing task.yaml. load_benchmark itself is torch-free — it only reads task.yaml and eval_spec.yaml — but raises ValueError for an unknown name/directory.

Benchmark metadata (always available)

These read the bundle’s committed task.yaml/eval_spec.yaml and never touch the manifest or case data:

bm.task_id    # "glioma_wt_tc_et" — task.yaml's task.id (falls back to the bundle dir name)
bm.modality   # "MRI"
bm.anatomy    # "brain"
bm.regions    # {"WT": [1, 2, 3], "TC": [1, 3], "ET": [3]} — eval_spec.yaml's eval_regions
bm.metrics    # ["dice", "hd95"]
bm.mode       # "3d_volumetric" — evaluate() only supports this mode today

Benchmark.train / Benchmark.test (manifest-gated)

bm.train   # list of train-split case ids
bm.test    # list of test-split case ids

These need the bundle’s manifest_test.json — a held-out asset kept out of the development source tree for most tasks. If it’s absent on your machine, both raise FileNotFoundError with a clear message.

Running volume-level evaluation

def evaluate(self, predictor, *, data_root, split="test", templates=None) -> ScoreReport

from medotter_bench.inference import ZeroPredictor

report = bm.evaluate(ZeroPredictor(), data_root="/path/to/DATA_ROOT", split="test")
print(report.macro)      # per-region mean, across the split's cases
print(report.per_case)   # {case_id: {region: {metric: value}}}

• predictor is a medotter_bench.inference.Predictor (predict(image) -> labels, where image is (C, i, j, k) — stacked channels over the volume’s native voxel grid, and labels is (i, j, k), normalized foreground ids) — not the 2D medotter.Predictor used by mo.evaluate. There is deliberately no generic 2D-model-to-volume adapter: running a 2D model over a volume needs per-modality preprocessing (CT HU windowing, slice axis, resampling) that only a task-specific predictor knows. Subclass medotter_bench.inference.BatchedSlicePredictor for that, or use ZeroPredictor (as above) to validate the path itself.
• data_root — the directory holding the bundle’s case files (NIfTI volumes, etc.), matched against the manifest by templates (a reference BraTS-format layout is the default; pass your own for a different DATA_ROOT layout).
• Only mode == "3d_volumetric" bundles are supported; anything else raises NotImplementedError.
• The returned ScoreReport.macro is a plain per-region mean — not the leaderboard’s case-weighted headline score. Treat it as a convenience summary, not the authoritative leaderboard number.

Common errors

• ValueError — load_benchmark got an unknown bundle name/directory, or evaluate’s split is not "train"/"test".
• FileNotFoundError — .train/.test/.evaluate need manifest_test.json, which is absent on this machine/bundle, or evaluate can’t find a case’s file under data_root.
• NotImplementedError — evaluate was called on a bundle whose .mode is not "3d_volumetric" (2D-image and instance-segmentation bundles are a follow-up).
• ImportError — evaluate pulls torch and nibabel (via medotter_bench.inference.run_submission) only when called; metadata access (.modality, .regions, etc.) never does.