Leaderboard

Submitting a model

Public package status: the Python distributions and desktop installers are not released yet. These pages document an evolving interface, not currently usable install artifacts. View release status.

Coming soon: Public model submissions are temporarily closed while we finish the production rollout. The leaderboard shows a disabled Submit Model control, and the public submit endpoint remains in maintenance mode. You can prepare and validate your Hugging Face repository now — everything below describes the submission flow that will go live, so you know what to expect.

Once your model repo satisfies the contract and is packaged correctly, it will be ready for the two launch paths described below. Both identify you via GitHub, POST to the same submit Supabase Edge Function, and poll the same submission_status view for progress — the difference is which client you use and when checks run.

There is no mo.submit()

This is deliberate. The medotter SDK’s job ends at local mo.evaluate(...) — scoring a model against a dataset, on your own machine. Submitting that model to the public leaderboard is a separate step: a CLI command (python -m medotter submit), not a Python library call. If you were looking for a function to import, it doesn’t exist. Prepare your repository with the pages in this section; the two submission paths below will become usable when public submissions open.

You need a GitHub account

Every submission — web or CLI — is tied to your GitHub identity. There is no anonymous or email-only path anymore. Concretely:

Your GitHub identity is never published on the leaderboard. It exists so the platform can (a) rate-limit fairly per account rather than per IP, and (b) give a human reviewer something to go on when moderating new submissions.

Rate limit: 1 submission per hour, per GitHub account

Each GitHub account may create at most one new submission per hour. A submission that exceeds this returns HTTP 429 with an exceeded_bucket field telling you which limit you hit (gh for your account, ip or global for site-wide protection) — the account limit (gh) is the one you’ll normally see. There’s nothing to do but wait; the window is a rolling hour, not a fixed clock hour.

Review, then run — with a 48-hour auto-release

A new submission does not go straight to the evaluation queue. It starts in awaiting review, and one of two things releases it from there:

  1. A maintainer approves it. This is expected to be the common case once volume is low; approval is quick since it’s just a sanity check before spending GPU time.
  2. 48 hours pass with no action. The submission auto-releases into the queue on its own — a stalled review never blocks you indefinitely. The leaderboard marks these rows with an auto badge so it’s visible that a human didn’t look at the model before it ran.

Either way, the model then runs through the exact same harness as every other submission. The platform’s own re-run is always the authoritative score — nothing you submit, including the self-report file below, ever substitutes for it.

Optional: attach a self-reported result file

If you already ran mo.evaluate(...) locally, you can attach a report of that run to your submission so a reviewer has something concrete to compare against the platform’s re-run — useful for catching an environment mismatch (a different CUDA version, a nondeterministic op) before it looks like a scoring bug.

Producing the report is the same either way: run mo.selfreport.evaluate_and_report(predictor, name, ...) locally, which writes a medotter-report.json file. What differs is how that file reaches your submission:

Either way, the file is JSON, capped at 256KB, and must be produced by the SDK — attaching a hand-edited file changes nothing about how you’re scored, since the platform’s re-run is authoritative; it just makes for a less useful report. Uploads over the cap or that aren’t valid JSON are rejected before anything is sent, so you’ll know immediately rather than after a round-trip.

Path A: the web Submit modal (coming soon)

The leaderboard currently shows a gray, disabled Submit Model — Coming soon control. After launch, it will open a form asking for:

The form validates the model id’s shape client-side (must look like org/model-name: exactly two non-empty segments, ASCII letters/digits/_.-, neither segment literally . or .., 200 characters max), the email’s shape if one is given, and the self-report file’s size/JSON validity if one is attached. It does not check that the repo exists, is public, or has a valid predictor.py — those checks happen server-side, after submission. Submitting requires being signed in with GitHub; until then, the button where Submit for evaluation would be reads Sign in with GitHub instead. Once submitted, a live progress view polls the submission’s status through review and evaluation until it reaches a terminal state.

Path B: the CLI (coming soon)

python -m medotter submit, part of the medotter SDK, is the recommended path — it combines the device-flow sign-in, an optional local self-report, and the submission POST in one command. The SDK installs no console scripts onto PATH; python -m runs the command under whichever interpreter you invoke it with — a stale console script on PATH can never point at a different environment:

python -m medotter submit owner/model \
  --revision <40-char-commit-sha> \
  --tasks glioma_wt_tc_et \
  --report medotter-report.json \
  --email you@example.com

Flags:

FlagRequiredMeaning
owner/modelyesHugging Face repo, positional argument.
--revisionyesThe exact 40-character commit SHA to submit (never a branch name — the platform pins to this commit).
--tasksyesComma-separated task ids, e.g. glioma_wt_tc_et. One submission can cover several.
--reportnoPath to a medotter-report.json produced by mo.selfreport.evaluate_and_report(...). Defaults to a same-named file in the current directory if one exists.
--emailnoContact email (optional).

What it does, in order:

  1. Device flow sign-in. Prints a verification_uri and a short code; approve it in your browser, and the CLI polls GitHub until you do.
  2. Resolve and pin. Confirms --revision resolves to a real commit and that predictor.py exists there.
  3. Attach the report, if given. Fills in the report’s submission identity (model, revision, predictor hash) to match what’s actually being submitted, then re-serializes it as the request’s self_report string.
  4. Submit. POSTs to {SUPABASE_URL}/functions/v1/submit with your GitHub token. On success (HTTP 201) prints the submission id; a 429 prints which rate-limit bucket you hit and to try again in an hour; any other non-2xx response prints the server’s error reason directly.

Credentials: the CLI needs no long-lived secret of your own — GitHub device-flow tokens are short-lived and scoped to the sign-in, and the project’s Supabase anon key (browser-safe, not a secret) ships with the SDK.

Which to use after launch

The CLI is the more complete path: it handles sign-in, pins the exact commit you tested, and attaches your self-report in one step, so it’s the recommended way to submit a model you’ve already evaluated locally. The web form is the friendlier entry point if you’re just looking at the leaderboard and want to try a model interactively, with visual progress — signing in with GitHub takes one click, and the self-report upload is there if you happen to have a report file handy.

Next: Bundle schema documents the on-disk task format your model is actually evaluated against.