Packaging your model
The leaderboard only ever downloads one file from your Hugging Face repo:
predictor.py, fetched unauthenticated from the repo root on its default
(main) branch:
https://huggingface.co/<owner>/<model>/resolve/main/predictor.py
That single constraint shapes the whole packaging story.
Repo layout
owner/model-name/ (Hugging Face model repo, public)
├── predictor.py ← required, at the repo root
├── model.pt ← your weights, any format/name you like
└── (anything else your load() needs, e.g. a config file)
predictor.pymust be at the repo root. It is fetched by a fixed.../resolve/main/predictor.pyURL — a nested path (src/predictor.py,models/predictor.py) will not be found.- Everything else is up to you. The platform never downloads your weights
directly.
load()is responsible for getting them — e.g. by reading a sibling file ifload()is running with your repo checked out, or by callinghuggingface_hub.hf_hub_download(repo_id=..., filename=...)to pull a weight file from your own repo at runtime. (This is the common pattern, not a documented requirement — the only hard requirement is thatload()ends up with a working predictor, however it gets there.) - The repo must be public. At submit time the platform’s submit function
fetches
predictor.pywith no auth (supabase/functions/submit/hf_artifact.ts, in theMedOtter-Webrepo) and rejects the submission if it can’t — a private or gated repo fails exactly this way. The eval worker’s own download (evaluation/eval_worker.py’s_download_predictor, in theMedOtter-Webrepo) useshuggingface_hub.hf_hub_download(repo_id=hf_model_id, filename="predictor.py")— the standard unauthenticated path for a public repo, run in the worker’s trusted parent process before the file is ever executed.
predictor.py is executed, not just parsed
Two different things happen to predictor.py, at two different times, and
it matters which one you’re thinking about:
- Submit time: fetched and pinned, never parsed. Nothing inspects your
code before it runs. The submit CLI (
medotter/cli.py, run aspython -m medotter submit) does no static analysis ofpredictor.pyat all — the old AST pre-flight (ascripts/submit_model.pythat parsed the file to confirm a top-levelload) no longer exists. At submit time the file is only fetched: the server’s submit function downloads it unauthenticated from.../resolve/<head>/predictor.py, rejects the submission if it’s missing (which is also what a private or gated repo looks like) or over 1 MiB, and pins its sha256. The CLI itself touches the file just once, computing that same sha256 (medotter/cli.py’sresolve_predictor_sha) when it attaches a self-report. Whether the file definesload— or even parses — is not checked until the worker executes it. - Real execution (the eval worker). When your submission actually runs,
the worker’s loader (
medotter_core.adapters.base.load_predictor) doesimportlib.util.spec_from_file_location(...)andspec.loader.exec_module(mod)on your downloadedpredictor.py, then callsmod.load(). This does execute every line of module-level code in your file, beforeload()is ever called.
The practical consequence: keep predictor.py importable and side-effect-free
at module scope.
- Importable — the file must be valid Python with no syntax errors and
must not raise on import. Neither is checked before the run: a
SyntaxError, aNameError, or a bad top-levelimportall surface only when the worker actually executes the file. - Side-effect-free at module scope — don’t load weights, open network
connections, or do heavy work at the top level of the file. Put that inside
load()(or inside your predictor’s__init__, called fromload()). Module-scope code runs once, immediately, as soon as the worker imports the file — before it has any chance to catch or report an error cleanly, and before yourload()function even gets a chance to run.
# Bad — model construction at import time; a failure here has nowhere to be
# reported cleanly, and it runs even if something downstream never calls load().
import torch
MODEL = torch.load("model.pt")
def load():
return MyPredictor(MODEL)
# Good — nothing but definitions at module scope; all the work happens
# inside load(), where a failure is attributable to "this submission's load()".
def load():
import torch
model = torch.load("model.pt")
return MyPredictor(model)
Optional platform hooks
Two helpers in medotter_core.inference let your predictor cooperate with
the worker’s batching and memory limits, if you want that — using them is
optional; a predictor that ignores both still works correctly, it just can’t
be batch-tuned or self-recover from an out-of-memory error:
batch_size_from_env(default)— returns the worker’sBATCH_SIZEenv var (set per-GPU, and halved on an OOM retry) if present and positive, else yourdefault. Route your model’s batch size through this so the platform can tune it per card.oom_safe_map(items, batch_size, fn, *, min_batch=1, on_oom=None)— runsfnoveritemsin chunks ofbatch_size, halving the batch size and clearing the CUDA cache in-process on a caught CUDA out-of-memory error, down tomin_batch, before re-raising (letting the worker’s own process-level retry be the final backstop).
BatchedSlicePredictor (see Contract) already
wires both of these in for 2D-slice models — you only need them directly if
your model has a different structure (whole-volume, 3D patches) and you’re
writing predict() yourself.
Next: Submitting covers how to point the leaderboard at your packaged repo.