Bundle schema

A bundle is a frozen benchmark task: a directory of files that is the single source of truth for which cases exist, how they’re split, and how predictions on them get scored. medotter_bench.bundle.Bundle is the loader both the website and the eval worker use to read one — its docstring describes a bundle directory as containing task.yaml, manifest_test.json, eval_spec.yaml, and coverage_report.{json,md}. Of those, Bundle itself only reads manifest_test.json eagerly (at construction) and task.yaml / eval_spec.yaml lazily (only when you call .task_spec() / .eval_spec()); coverage_report.* is documentary — human/website-facing coverage notes, never read by Bundle’s own API.

Those four files don’t all live in one place. The medotter SDK repo’s benchmarks/<name>/ directories are the factory side: newer bundles carry task.yaml and eval_spec.yaml (older ones like vessel_ct predate that convention — see the task.yaml section below), while manifest_test.json is only git-tracked for some bundles (vessel_ct, thoracic_lesion_ct, histopath_nucleus_seg, nucleus_instance_2d, nucleus_instance_3d) — others, like glioma_wt_tc_et, build their manifest from data that isn’t checked into that repo. This website repo (MedOtter-Web) is the consumer side: it vendors full, frozen copies of each bundle under evaluation/bundles/<name>/, and the vendored copy may additionally include coverage_report.{json,md} — currently only evaluation/bundles/glioma_wt_tc_et/ ships them, alongside task.yaml, eval_spec.yaml, and manifest_test.json (the other vendored bundles carry just the first three). That vendored copy is what the eval worker actually loads at runtime.

This page documents the real field names, read directly from medotter_bench/bundle.py and the bundle directories under medotter/benchmarks/ in the SDK repo — not invented.

Bundle

from medotter_bench.bundle import Bundle

# thoracic_lesion_ct: one of the bundles whose manifest_test.json (and
# task.yaml, and eval_spec.yaml) is actually git-tracked in the medotter repo.
# (glioma_wt_tc_et's manifest_test.json is NOT tracked there — Bundle() eagerly
# reads it at construction, so pointing this example at glioma would raise
# FileNotFoundError before you got to call anything.)
b = Bundle("benchmarks/thoracic_lesion_ct")
b.task      # manifest["task"]     -> "thoracic_lesion_ct"
b.version   # manifest["version"]
b.hash      # manifest["hash"]
b.train     # manifest["train"]    -> list of case ids
b.test      # manifest["test"]     -> list of case ids
b.case_meta("COVID_19_20/volume-covid19-A-0003")   # manifest["per_case"][case_id]
list(b.cases())                          # (case_id, meta) for every case, train+test
b.task_spec()   # yaml.safe_load(task.yaml)   — lazy, re-parsed each call
b.eval_spec()    # yaml.safe_load(eval_spec.yaml) — lazy, re-parsed each call

task.yaml

The task ontology: what the benchmark measures, its canonical label space, and (the real, populated remap mechanism) the per-dataset native→canonical label id map. Real fields, grounded in benchmarks/thoracic_lesion_ct/task.yaml and benchmarks/abdominal_multiorgan_ct/task.yaml:

• task.id / task.name / task.modality / task.anatomy / task.description — identity block: id matches the bundle directory name and manifest_test.json’s task field; name is a human-readable title; modality is the imaging modality (CT, MRI, …); anatomy is the region scored; description is free text on what’s measured and how.

• canonical_labels — {region_name: [canonical_label_ids]}, the same shape eval_spec.yaml’s eval_regions uses. A region can be a single id (tumor_nodule: [1]) or a union (glioma’s WT: [1, 2, 3]).

• main_region — the headline region name, or macro_avg for a macro-average across regions.

• label_remap — the mechanism that actually remaps ground truth today. Per-dataset (not per-case) native→canonical label id map: keys are dataset names, values are {native_id: canonical_id, ...} (a note key is documentary and stripped by the SDK). Any native id present in a dataset’s raw mask but not listed in that dataset’s map drops to background (0) — it’s excluded from scoring, not counted as a miss. Applied by score_predictions to the ground truth (loaded in native space by medotter_bench.inference.VolumeLoader) before scoring; predictions must already be in canonical space (eval_spec.yaml’s prediction_format).

  This {native_id: canonical_id} dict shape is what thoracic_lesion_ct and abdominal_multiorgan_ct use; it isn’t universal. glioma_wt_tc_et instead records {native: [...], to: [...]} lists per dataset — its dataloaders pre-normalize labels before the bundle ever sees them, so that form is documentary rather than applied at scoring time. And not every bundle has a task.yaml at all: vessel_ct predates this spec-as-code convention and does its native→canonical remap in a bundle-local remap.py / scoring.py instead.

# benchmarks/thoracic_lesion_ct/task.yaml — trimmed
task:
  id: thoracic_lesion_ct
  name: Unified Thoracic Lesion Segmentation (CT)
  modality: CT
  anatomy: thorax / lung
canonical_labels:
  tumor_nodule: [1]
  infection:    [2]
main_region: macro_avg
label_remap:
  NSCLC_Radiomics_Interobserver1: {1: 1}
  MIDRC_RICORD_1a: {1: 2, 2: 2, 3: 2}   # native 4/5/6, not listed, drop to bg

manifest_test.json

The frozen per-case file listing and train/test split. Required keys, per Bundle’s accessors: task, version, hash, train, test, per_case. Real manifests in this repo also carry split_version, and the newer ones add skipped / dropped_cross_split bookkeeping arrays — those aren’t read by Bundle, but they’re part of the real on-disk shape.

{
  "task": "vessel_ct",
  "version": "0.1.0",
  "split_version": "vessel_ct/v1",
  "hash": "facbb5f45b72",
  "train": ["FLARE22/FLARE22_Tr_0001", "..."],
  "test": ["BTCV/0001", "..."],
  "per_case": {
    "BTCV/0001": {
      "dataset": "BTCV",
      "split": "test",
      "tier": "test",
      "patient_uid": "0001",
      "image": "btcv/RawData/Training/img/img0001.nii.gz",
      "seg": "btcv/RawData/Training/label/label0001.nii.gz"
    }
  }
}

per_case[case_id] fields, observed across bundles in this repo (the exact set varies by bundle/generation):

• dataset — which source dataset this case came from (e.g. "BTCV", "FLARE22").
• split — "train" or "test".
• tier / region — older bundles (e.g. vessel_ct) carry tier (e.g. "train" / "near_ood" / "far_ood"); newer instance-segmentation bundles (e.g. nucleus_instance_2d) carry region instead (the canonical region name this case scores, e.g. "nucleus").
• patient_uid / series_uid — patient/series identifiers (newer bundles); may be null when not applicable.
• image — path to the input, relative to data_root. Used for single-channel inputs (CT, RGB tiles).
• chan — for multi-channel inputs (e.g. multi-parametric MRI), an ordered {channel_name: relpath} map instead of image; the model’s input channel order follows this map’s order — the manifest is the single source of truth for channel order, per medotter_bench.inference.VolumeLoader.
• seg — path to the ground-truth mask, relative to data_root. Never staged into the untrusted inference sandbox.
• remap — an optional per-case native→canonical label id remap slot (present in the field’s docs, e.g. medotter_bench/inference.py’s {4: 3} example). It’s currently unpopulated by every manifest in this repo — medotter_bench/inference.py notes manifests “don’t carry a per-case remap yet”, so reading it is a no-op today. The remap that’s actually applied is a level up, in task.yaml’s label_remap (see above) — a per-dataset, not per-case, native→canonical map.

All manifest-supplied paths are resolved relative to data_root (or its _prepared/ subdirectory when present) and validated against absolute paths and .. traversal before being opened.

eval_spec.yaml

The scoring protocol: which regions/classes are scored, which metrics, and (for bundles that declare it) which adapter runs the scoring. Real fields, observed across bundles in this repo:

• task_type — dispatch key into the adapter registry (see below). Optional — if absent, medotter_bench.engine.resolve_task_type defaults it to "seg", specifically so pre-existing bundles (glioma, abdominal, thoracic, vessel) needed no edit when the generic task_type-dispatch engine was added.
• mode — documentary description of the scoring granularity, e.g. 3d_volumetric (per-patient volume) or 2d_image (per-tile).
• segmentation_mode — documentary, e.g. multi_class or instance.
• eval_regions — {region_name: [canonical_label_ids]}. A region can be a single id ({tumor_nodule: [1]}) or a union of several ({WT: [1, 2, 3]} for “whole tumor”).
• partial_label — true/false. When true, scoring for a given test case is restricted to the region(s) that case’s dataset actually labels (unlabeled regions are excluded from scoring entirely, not counted as misses).
• main_region — the headline region name, or macro_avg for a macro-average across regions.
• metrics — list of metric names to compute. Semantic segmentation tasks use [dice, hd95]; instance segmentation tasks use [f1, aji, pq] (F1@IoU, mean Aggregated Jaccard Index, Panoptic Quality) and never dice/hd95.
• hd95 — sub-object (percentile, spacing, unit) configuring the 95th-percentile Hausdorff distance metric, present only on tasks that score it.
• iou_threshold — instance-segmentation tasks only: the IoU at which a predicted instance counts as a true-positive match to a ground-truth instance.
• empty_structure — per-metric fixed-penalty table for the GT-empty/prediction-empty edge cases (e.g. gt_nonempty_pred_empty: 373.13 mm for HD95 — a missed structure is penalized at a fixed cap, not excluded from scoring).
• aggregation — how per-case scores roll up: unit (e.g. patient or image), reduce (e.g. mean), report_by_region / report_by_organ / report_by_dataset / report_by_tier (booleans/lists controlling breakdown reporting — report_by_organ is abdominal_multiorgan_ct’s name for the same per-region breakdown other bundles call report_by_region), and headline: {region, metric[, tier]} naming the single number the leaderboard’s headline column shows.
• prediction_format — documents what your prediction must look like: type (e.g. nifti or instance_mask2d), label_space (the canonical id set, e.g. [0, 1, 2, 3]), coordinate_space (e.g. input_volume_grid — same grid as the input, no reorientation), and a free-text note.

Two real examples, trimmed:

# benchmarks/thoracic_lesion_ct/eval_spec.yaml — semantic seg, task_type
# absent (implicitly "seg")
mode: 3d_volumetric
segmentation_mode: multi_class
eval_regions:
  tumor_nodule: [1]
  infection: [2]
partial_label: true
main_region: macro_avg
metrics: [dice, hd95]

# benchmarks/nucleus_instance_2d/eval_spec.yaml — instance seg, task_type
# explicit
task_type: instance_seg_2d
mode: 2d_image
segmentation_mode: instance
eval_regions:
  nucleus: [1]
partial_label: false
main_region: nucleus
metrics: [f1, aji, pq]
iou_threshold: 0.5

Task types

Registered in medotter_bench/adapters/registry.py, populated by medotter_bench/adapters/__init__.py. Implemented (a submission can actually be scored against these):

Also registered, but not implemented — stub adapters that raise NotImplementedError if a bundle declares them: cls, det, vlm.

Next: this closes the submitter track. See Contract for what your model needs to expose, and Guides → Evaluation for the SDK-side scoring API these bundles share metrics with.