Submitting a model
Coming soon: Public model submissions are temporarily closed while we finish the production rollout. The leaderboard shows a disabled Submit Model control, and the public submit endpoint remains in maintenance mode. You can prepare and validate your Hugging Face repository now — everything below describes the submission flow that will go live, so you know what to expect.
Once your model repo satisfies the contract and
is packaged correctly, it will be ready for the
two launch paths described below. Both identify you via GitHub, POST to the
same submit Supabase Edge Function, and poll the same submission_status
view for progress — the difference is which client you use and when checks
run.
There is no mo.submit()
This is deliberate. The medotter SDK’s job ends at local
mo.evaluate(...) — scoring a model against a dataset, on your own machine.
Submitting that model to the public leaderboard is a separate step: a CLI
command (python -m medotter submit), not a Python library call. If you were looking
for a function to import, it doesn’t exist. Prepare your repository with the
pages in this section; the two submission paths below will become usable when
public submissions open.
You need a GitHub account
Every submission — web or CLI — is tied to your GitHub identity. There is no anonymous or email-only path anymore. Concretely:
- Web: the Submit Model modal asks you to sign in with GitHub (via Supabase Auth) before it will let you submit. Filling out the form doesn’t require signing in first, but submitting does — the button reads Sign in with GitHub until you do.
- CLI:
python -m medotter submitwalks you through GitHub’s OAuth device flow — it prints a URL and a short code, you approve it in your browser, and the CLI picks up from there. No password or token to copy around.
Your GitHub identity is never published on the leaderboard. It exists so the platform can (a) rate-limit fairly per account rather than per IP, and (b) give a human reviewer something to go on when moderating new submissions.
Rate limit: 1 submission per hour, per GitHub account
Each GitHub account may create at most one new submission per hour. A
submission that exceeds this returns HTTP 429 with an exceeded_bucket
field telling you which limit you hit (gh for your account, ip or
global for site-wide protection) — the account limit (gh) is the one
you’ll normally see. There’s nothing to do but wait; the window is a rolling
hour, not a fixed clock hour.
Review, then run — with a 48-hour auto-release
A new submission does not go straight to the evaluation queue. It starts in awaiting review, and one of two things releases it from there:
- A maintainer approves it. This is expected to be the common case once volume is low; approval is quick since it’s just a sanity check before spending GPU time.
- 48 hours pass with no action. The submission auto-releases into the queue on its own — a stalled review never blocks you indefinitely. The leaderboard marks these rows with an auto badge so it’s visible that a human didn’t look at the model before it ran.
Either way, the model then runs through the exact same harness as every other submission. The platform’s own re-run is always the authoritative score — nothing you submit, including the self-report file below, ever substitutes for it.
Optional: attach a self-reported result file
If you already ran mo.evaluate(...) locally, you can attach a report of
that run to your submission so a reviewer has something concrete to compare
against the platform’s re-run — useful for catching an environment mismatch
(a different CUDA version, a nondeterministic op) before it looks like a
scoring bug.
Producing the report is the same either way: run
mo.selfreport.evaluate_and_report(predictor, name, ...) locally, which
writes a medotter-report.json file. What differs is how that file reaches
your submission:
- Recommended: run
python -m medotter submitfrom the same directory — it automatically picks upmedotter-report.json(or a path you pass via--report) and attaches it as part of submitting (see below). The CLI does not run the evaluation itself; it only attaches the filemo.selfreport.evaluate_and_report(...)already produced. - Also fine: attach that same file yourself in the web modal’s optional file field.
Either way, the file is JSON, capped at 256KB, and must be produced by the SDK — attaching a hand-edited file changes nothing about how you’re scored, since the platform’s re-run is authoritative; it just makes for a less useful report. Uploads over the cap or that aren’t valid JSON are rejected before anything is sent, so you’ll know immediately rather than after a round-trip.
Path A: the web Submit modal (coming soon)
The leaderboard currently shows a gray, disabled Submit Model — Coming soon control. After launch, it will open a form asking for:
- HuggingFace Model ID (required) — text input, e.g.
org/model-name. - Contact email (optional) — a way to reach you if a reviewer has a question; it is never used for rate-limiting or identity.
- Self-reported results (optional) — the SDK-produced JSON file described above.
- Tasks to evaluate (required, at least one) — a checklist of tasks grouped by modality; check individual tasks or a whole modality group.
The form validates the model id’s shape client-side (must look like
org/model-name: exactly two non-empty segments, ASCII letters/digits/_.-,
neither segment literally . or .., 200 characters max), the email’s shape
if one is given, and the self-report file’s size/JSON validity if one is
attached. It does not check that the repo exists, is public, or has a
valid predictor.py — those checks happen server-side, after submission.
Submitting requires being signed in with GitHub; until then, the button
where Submit for evaluation would be reads Sign in with GitHub
instead. Once submitted, a live progress view polls the submission’s status
through review and evaluation until it reaches a terminal state.
Path B: the CLI (coming soon)
python -m medotter submit, part of the medotter SDK, is the recommended
path — it combines the device-flow sign-in, an optional local self-report,
and the submission POST in one command. The SDK installs no console scripts
onto PATH; python -m runs the command under whichever interpreter you
invoke it with — a stale console script on PATH can never point at a
different environment:
python -m medotter submit owner/model \
--revision <40-char-commit-sha> \
--tasks glioma_wt_tc_et \
--report medotter-report.json \
--email you@example.com
Flags:
| Flag | Required | Meaning |
|---|---|---|
owner/model | yes | Hugging Face repo, positional argument. |
--revision | yes | The exact 40-character commit SHA to submit (never a branch name — the platform pins to this commit). |
--tasks | yes | Comma-separated task ids, e.g. glioma_wt_tc_et. One submission can cover several. |
--report | no | Path to a medotter-report.json produced by mo.selfreport.evaluate_and_report(...). Defaults to a same-named file in the current directory if one exists. |
--email | no | Contact email (optional). |
What it does, in order:
- Device flow sign-in. Prints a
verification_uriand a short code; approve it in your browser, and the CLI polls GitHub until you do. - Resolve and pin. Confirms
--revisionresolves to a real commit and thatpredictor.pyexists there. - Attach the report, if given. Fills in the report’s submission identity
(model, revision, predictor hash) to match what’s actually being
submitted, then re-serializes it as the request’s
self_reportstring. - Submit. POSTs to
{SUPABASE_URL}/functions/v1/submitwith your GitHub token. On success (HTTP 201) prints the submission id; a429prints which rate-limit bucket you hit and to try again in an hour; any other non-2xx response prints the server’s error reason directly.
Credentials: the CLI needs no long-lived secret of your own — GitHub device-flow tokens are short-lived and scoped to the sign-in, and the project’s Supabase anon key (browser-safe, not a secret) ships with the SDK.
Which to use after launch
The CLI is the more complete path: it handles sign-in, pins the exact commit you tested, and attaches your self-report in one step, so it’s the recommended way to submit a model you’ve already evaluated locally. The web form is the friendlier entry point if you’re just looking at the leaderboard and want to try a model interactively, with visual progress — signing in with GitHub takes one click, and the self-report upload is there if you happen to have a report file handy.
Next: Bundle schema documents the on-disk task format your model is actually evaluated against.