Evaluation

Two entry points cover both ends of the evaluation workflow: evaluate runs a model over a dataset and scores it in one call; score takes predictions you already have and scores them against a loaded dataset. Both return the same ScoreReport.

The Predictor protocol

evaluate drives any object satisfying this protocol (it’s runtime_checkable, so isinstance(model, mo.Predictor) works, but nothing needs to subclass it — duck typing is enough):

class Predictor(Protocol):
    def predict(self, image: np.ndarray, spacing=None) -> np.ndarray: ...

predict returns an integer label map matching image’s spatial shape, in the dataset’s native label space (see Sample). A bare callable model(image) — with no .predict method — also works with evaluate, called positionally.

Scoring predictions you already have: score

def score(predictions: dict, dataset, *, metrics=("dice", "iou"),
          labels=None, schema=None, per_case=False, present_only=False) -> ScoreReport

• predictions — {key: pred_label_array}. key is f"{case_id}#{slice_idx}" when a sample’s meta has both, else case_id, else the sample’s index as a string — the same rule evaluate uses internally.
• dataset — a mo.load_dataset(...) view, or any iterable of Sample.
• metrics — any of "dice", "iou", "hd95".
• labels / schema — see below; pass at most one.
• per_case — when True, ScoreReport.per_case also carries each case’s individual metric values.
• present_only — when True, a class’s per-case mean is averaged only over the cases where that class is present in the ground truth (by default, GT-absent cases are still included at their “empty structure” score: 1.0 if the prediction is also empty for that class, 0.0 on a false positive — matching the leaderboard convention).

import medotter as mo

ds = mo.load_dataset("ACDC", split="test")
preds = {"patient001_frame01": my_prediction_array, ...}  # keyed like _case_key
report = mo.score(preds, ds, metrics=("dice", "iou"), per_case=True)

One call, model included: evaluate

def evaluate(model, name: str, *, split="test", type="image",
             metrics=("dice", "iou"), labels=None, schema=None,
             per_case=False, present_only=False, **load_kwargs) -> ScoreReport

evaluate loads name via load_dataset (forwarding split, type, and any **load_kwargs), runs model over every sample, and scores the predictions with the same logic score uses.

model = mo.load_model("sam2_hiera_small")
report = mo.evaluate(model, "ACDC", split="test", metrics=("dice", "iou"))
print(report)   # per-class dice/iou + macro average

Use evaluate when you want the model run for you; use score when the predictions already exist — computed separately, cached from a previous run, or produced by a model medotter doesn’t know how to load.

Label semantics: labels= vs schema=

By default, label semantics (which ids are background, which are ignored, what each foreground id is named) are resolved from the dataset’s catalog entry, or inferred from the ground-truth masks if the catalog has no class info. Two ways to override that:

• labels= — a list of class names (index 0 conventionally background) or a {id: name} dict. The backwards-compatible, more permissive option; a dict explicitly naming id 0 opts that id into scoring (for zero-indexed- foreground datasets like WSSS4LUAD).
• schema= — an explicit medotter.LabelSchema(classes, background_id=0, ignore_ids=frozenset()). The strict, exhaustive contract: any ground-truth or prediction id outside the schema raises ValueError rather than being silently dropped.

Passing both labels and schema raises ValueError.

ScoreReport

@dataclass(frozen=True)
class ScoreReport:
    per_class: dict[str, dict[str, float | None]]
    macro: dict[str, float | None]
    n_cases: int
    metrics: tuple[str, ...]
    support: dict[str, int]
    per_case: dict[str, dict[str, dict[str, float]]] | None

per_class holds each foreground class’s mean per metric; macro averages those means over classes that are present in at least one ground-truth case (a class never present anywhere is reported in per_class/support but excluded from macro). support gives each class’s GT-present case count. per_case is populated only when per_case=True was passed.

Common errors

• KeyError — score() has no entry in predictions for one of the dataset’s case keys.
• ValueError — the number of predictions doesn’t match the number of samples; a duplicate case key was detected (a volume-level case_id shared across samples with no slice_idx); both labels and schema were passed; an unsupported metric name was requested; or a prediction contains a label id outside the resolved schema.
• ImportError — score/evaluate import the torch-free metrics package on first call, which pulls in scipy eagerly (it backs hd95) regardless of which metrics= you asked for; a missing scipy surfaces here. evaluate additionally needs the data stack load_dataset requires.