Rules & evaluation

How submissions are limited, scored and ranked.

Deadline

Submissions close Dec 11, 2026 11:59 PM PST ( left). Uploads after the deadline are refused.

Submission limits

  • At most 5 accepted submissions per team in any rolling 24-hour window.
  • Every accepted submission counts, even if its evaluation later fails or times out. Uploads rejected by the validator (wrong format, missing files) do not count.
  • One evaluation per team at a time: wait for your current submission to finish before uploading the next.
  • One evaluation runs on the server at a time, in submission order, so queue times grow near the deadline.
  • The instructor can adjust limits for individual teams and can disable submissions that break the rules.

Scoring

The hidden test set is split once, before the competition, into a public subset (50%) and a private subset (50%), stratified by label. Your script predicts every row without knowing which subset it belongs to.

  • Primary metric: AUPRC (average precision). Threshold metrics use a cut-off of 0.5. How to compute AUPRC yourself.
  • The public leaderboard shows each team's best public score. Equal scores: the team with fewer submissions ranks higher; if that is also equal, the team whose best submission came earlier ranks higher. The final ranking breaks ties the same way.
  • Final ranking: each team's best public submission is scored on the private subset, and teams are ranked by that score. The best private score among your other attempts is never used.
  • The instructor may freeze the public leaderboard near the end. The private leaderboard is revealed when final results are published.

Fair play

  • Do not try to obtain hidden labels, probe the evaluation system, access other teams' files, or attack the server. Every submission runs in an isolated sandbox with no network, and every artifact is stored with its hash.
  • Do not hard-code predictions or try to identify the anonymised blind-test companies.
  • One team per group. Keep your access token private; it is the only way to sign in as your team.
  • Violations lead to disqualified submissions or teams.