medotter_bench

medotter_bench is the torch-free package behind the leaderboard’s scoring worker: the frozen-bundle loader, the prompt-free inference contract, and the scoring harness. Importing it pulls in scipy (via metrics.hd95) but never torch — the submitted model brings its own. medotter_bench.__version__ is "0.6.0".

This page documents:

• The submitter-facing inference surface — Predictor, BatchedSlicePredictor, VolumeLoader (importable only from the medotter_bench.inference submodule), plus run_submission (which, unlike the other three, is in the top-level __all__ and importable directly from medotter_bench).
• Scoring exports relevant to submitters from the top-level medotter_bench.__all__ — Bundle, score_regions, region_dice, region_iou, hd95, evaluate, resolve_task_type.
• Everything else in __all__, briefly, for completeness.

If you’re writing a predictor.py for a Hugging Face submission, start with the leaderboard submitter track instead — this page is the terse API reference, not a tutorial.

The submitter-facing inference surface (medotter_bench.inference)

Import-path caveat: Predictor, BatchedSlicePredictor, and VolumeLoader are defined in — and only importable from — the medotter_bench.inference submodule. None of these three are re-exported in medotter_bench’s top-level __all__:

# Fails: ImportError — not in medotter_bench.__all__
from medotter_bench import Predictor

# Works:
from medotter_bench.inference import Predictor, BatchedSlicePredictor, VolumeLoader

run_submission is different: it is listed in medotter_bench.__all__ (see medotter_bench/__init__.py), so importing it directly from medotter_bench works and is the idiomatic form — same as the other documented top-level functions (Bundle, score_regions, evaluate, resolve_task_type, etc.):

# Also works — run_submission is in medotter_bench.__all__:
from medotter_bench import run_submission

The three-class caveat is deliberate: Predictor/BatchedSlicePredictor/ VolumeLoader are documentation/convenience for people writing a submission (see the submission contract — the worker’s actual contract is duck-typed, load() need only return an object with .predict(image)), not part of the package’s own scoring API surface. medotter_bench.inference also exports ZeroPredictor (a trivial all-background baseline), InstanceVolumeLoader/ImageLoader/ InstanceImageLoader (loaders for the 2D and instance-segmentation task types), and the batching helpers batch_size_from_env/oom_safe_map — not detailed on this page; see medotter_bench/inference.py.

Predictor

class Predictor:
    def predict(self, image: np.ndarray) -> np.ndarray: ...

Stability: stable

The submission contract’s reference base class. Subclass and implement predict; subclassing is optional (see the duck-typing note) — any object with a matching .predict satisfies the worker.

Returns: a label array matching image’s spatial shape (no channel axis), integer ids in the task’s canonical label space (e.g. {0,1,2,3} for the glioma WT/TC/ET task).

Raises: NotImplementedError if called on the base class directly (the base predict is a stub — subclasses must override it).

from medotter_bench.inference import Predictor

class MyPredictor(Predictor):
    def predict(self, image):
        return my_model(image)   # (C, i, j, k) -> (i, j, k) label volume

BatchedSlicePredictor

class BatchedSlicePredictor(Predictor):
    default_batch = 8
    def units(self, image): ...
    def infer_batch(self, units): ...
    def combine(self, image, outputs): ...

Stability: stable

An optional reference base for 2D-slice models: implement three hooks and predict() is provided for you, reading the worker’s BATCH_SIZE env var (via batch_size_from_env) and halving the batch in-process on a CUDA out-of-memory error (via oom_safe_map) before falling back to the worker’s own process-level retry.

Returns (predict): the label volume/image combine returns.

Raises: NotImplementedError for any hook not overridden; propagates a CUDA OOM RuntimeError if it persists after backing off to min_batch=1.

Models with a non-slice structure (3D patches, whole-volume inference) should call batch_size_from_env + oom_safe_map directly inside their own predict() instead of subclassing this.

from medotter_bench.inference import BatchedSlicePredictor

class MySlicePredictor(BatchedSlicePredictor):
    default_batch = 16
    def units(self, image):
        return [image[:, :, :, k] for k in range(image.shape[-1])]
    def infer_batch(self, units):
        return [my_model(u) for u in units]
    def combine(self, image, outputs):
        return np.stack(outputs, axis=-1)

VolumeLoader

class VolumeLoader:
    def __init__(self, data_root, templates=None, prefer_prepared=True): ...

Stability: stable

Maps a bundle case to (image[C,i,j,k], gt[i,j,k], spacing) via nibabel, in native voxel order (no canonical reorientation). Used internally by run_submission/run_inference/Benchmark.evaluate; you generally don’t construct it directly unless writing a custom inference driver.

Key methods: load_input(case_id, meta) -> np.ndarray (the (C,i,j,k) model input); load_gt(case_id, meta) -> (np.ndarray, spacing) (ground truth + voxel spacing — never staged into an untrusted inference sandbox); stage_paths(case_id, meta) -> (input_paths, gt_path).

Raises: ValueError if a manifest-supplied path is absolute or escapes data_root (path-traversal guard); FileNotFoundError/KeyError from nibabel/dict lookups for a malformed case entry.

from medotter_bench.inference import VolumeLoader

loader = VolumeLoader("/path/to/DATA_ROOT")
image = loader.load_input("BraTS2023-GLI_00001", {"dataset": "BraTS2023-GLI"})

run_submission

def run_submission(predictor, data_root, bundle_dir, split="test", templates=None)

Stability: stable

Convenience full-pipeline (inference + scoring) for offline REAL/SMOKE runs and tests — streams per case, sharing one VolumeLoader so it never holds every predicted volume in memory at once. The production worker instead runs inference (untrusted child, GT-free) and scoring (trusted parent) in separate processes; run_submission is for local/offline use, not what the worker calls in production.

Unlike Predictor/BatchedSlicePredictor/VolumeLoader above, run_submission is in medotter_bench.__all__ — from medotter_bench import run_submission works directly.

Returns: {case_id: {region: {metric: value}}}.

Raises: propagates whatever predictor.predict / nibabel loading raises for a bad case; FileNotFoundError if bundle_dir has no manifest_test.json.

from medotter_bench import run_submission
from medotter_bench.inference import ZeroPredictor

per_case = run_submission(ZeroPredictor(), "/path/to/DATA_ROOT",
                          "benchmarks/thoracic_lesion_ct", split="test")
per_case["COVID_19_20/volume-covid19-A-0003"]   # {'infection': {'dice': ..., 'hd95': ...}} — partial-label: only the region this dataset labels is scored

Scoring exports

The following are all re-exported in medotter_bench.__all__ — importable directly as from medotter_bench import ....

Bundle

from medotter_bench.bundle import Bundle

class Bundle:
    def __init__(self, root): ...

Stability: stable

Loads a frozen benchmark bundle: the factory-repo ↔ website contract. Reads manifest_test.json eagerly at construction; task.yaml/eval_spec.yaml lazily, only when .task_spec()/.eval_spec() are called.

Accessors (read from medotter_bench/bundle.py):

Raises: FileNotFoundError if root has no manifest_test.json (raised at construction, not first use); KeyError from the property accessors if a required manifest key is missing.

See the Bundle schema page for the real field names inside task.yaml, manifest_test.json, and eval_spec.yaml.

from medotter_bench.bundle import Bundle

b = Bundle("benchmarks/thoracic_lesion_ct")
b.task, b.train[:2], b.test[:2]

score_regions

def score_regions(pred_label: np.ndarray, gt_label: np.ndarray,
                  spacing=(1.0, 1.0, 1.0), regions=CANONICAL_REGIONS,
                  metrics=("dice", "hd95")) -> dict

Stability: stable

Scores every region as the union of its constituent label ids, for a full label map (not a pre-binarized mask).

Returns: {region_name: {metric: value}}, e.g. {'WT': {'dice': 0.91, 'hd95': 3.2}, 'TC': {...}, 'ET': {...}}.

Raises: nothing explicit; propagates numpy shape errors if pred_label and gt_label don’t match.

from medotter_bench import score_regions

score_regions(pred_labels, gt_labels, spacing=(1.0, 1.0, 1.0), metrics=("dice", "iou"))

region_dice

def region_dice(pred_bin: np.ndarray, gt_bin: np.ndarray) -> float

Stability: stable

Binary Dice over two already-binarized region masks.

Params: pred_bin, gt_bin — boolean (or 0/1) arrays of the same shape, one region already unioned from a label map.

Returns: a float in [0, 1]. Empty-structure convention: both masks empty → 1.0; exactly one empty → 0.0.

Raises: nothing explicit.

from medotter_bench import region_dice

region_dice(pred_wt_mask, gt_wt_mask)

region_iou

def region_iou(pred_bin: np.ndarray, gt_bin: np.ndarray) -> float

Stability: stable

Binary IoU (Jaccard) over two already-binarized region masks, with the same empty-structure convention as region_dice. Torch-free — this is a different implementation from the repo’s torch-based metrics.iou.iou_score.

Params/Returns/Raises: same shape as region_dice.

from medotter_bench import region_iou

region_iou(pred_wt_mask, gt_wt_mask)

hd95

def hd95(pred: np.ndarray, gt: np.ndarray, spacing=(1.0, 1.0, 1.0)) -> float

Stability: stable

95th-percentile symmetric Hausdorff distance (mm), between two binary masks.

Returns: a float (mm). Empty-structure convention: both empty → 0.0; exactly one empty → HD95_MAX_MM (373.13, the BraTS convention — a fixed penalty, not dropped from the mean).

Raises: nothing explicit; needs scipy (imported eagerly by metrics.hd95 on first import of anything in medotter_bench).

from medotter_bench import hd95

hd95(pred_wt_mask, gt_wt_mask, spacing=(1.0, 1.0, 1.5))

evaluate

def evaluate(bundle_dir, artifact, data_root, split="test") -> dict

Stability: stable

Not the same function as medotter.evaluate (the 2D facade helper) or Benchmark.evaluate (the medotter.load_benchmark(...) method) — this is medotter_bench.engine.evaluate, a one-process load+infer+score helper for local/offline use, dispatching to the bundle’s registered adapter (see resolve_task_type below). The leaderboard worker does not call this — it runs the adapter’s inference and scoring halves in separate processes so untrusted submission code never sees ground truth.

Returns: the adapter’s evaluation result (shape depends on the bundle’s task_type; for seg, {case_id: {region: {metric: value}}}).

Raises: whatever the resolved adapter’s evaluate raises; NotImplementedError for a bundle whose task_type resolves to a registered-but-unimplemented stub adapter (cls, det, vlm — registered in medotter_bench/adapters/stubs.py, whose load/gt_loader/infer_iter/score all raise); AttributeError if artifact is not a dict (e.g. a bare Predictor instance); ValueError if artifact is a dict that matches none of the three forms above. (A task_type with no adapter registered at all — not even a stub — raises medotter_bench.adapters.registry.UnknownTaskType instead, but every currently-defined task_type has at least a stub.)

from medotter_bench import evaluate

report = evaluate("benchmarks/thoracic_lesion_ct", {"smoke": True}, "/path/to/DATA_ROOT")

# or, passing your own Predictor instance:
from medotter_bench.inference import ZeroPredictor
report = evaluate("benchmarks/thoracic_lesion_ct", {"predictor": ZeroPredictor()},
                  "/path/to/DATA_ROOT", split="test")

resolve_task_type

def resolve_task_type(bundle_dir) -> str

Stability: stable

Returns: the bundle’s task_type from eval_spec.yaml — one of "seg" (default, when absent), "seg2d", "instance_seg_2d", "instance_seg_3d", "prompt_seg", or a not-yet-implemented "cls"/"det"/"vlm". Defaulting to "seg" means pre-existing bundles (glioma, abdominal, thoracic, vessel) needed no edit when task-type dispatch was introduced.

Raises: FileNotFoundError if bundle_dir has no manifest_test.json (via the Bundle it constructs internally).

from medotter_bench import resolve_task_type

resolve_task_type("benchmarks/nucleus_instance_2d")   # "instance_seg_2d"
resolve_task_type("benchmarks/thoracic_lesion_ct")     # "seg" (task_type absent -> default)

Other top-level exports

The rest of medotter_bench.__all__ is primarily used by the eval worker itself rather than by submitters; documented briefly here for completeness (see medotter_bench/harness.py, medotter_bench/inference.py, and medotter_bench/engine.py for full detail):