How to submit

Everything your submission needs: the data, the predict.py interface, the output format and the evaluation environment.

1. The task

Predict AAER_related: 1 if a company-fiscal-year is linked to an SEC Accounting and Auditing Enforcement Release, 0 for a matched control firm-year. About one row in four is positive, so accuracy is misleading; the competition is scored by AUPRC (average precision) on predicted probabilities.

File (in student_data/)Use
train.csv, validation.csvLabelled firm-years for training and model selection (label column AAER_related).
blind_test_features.csvBlind test A — unseen companies. Features only.
future_blind_test_features.csvBlind test B — fiscal years 2019–2021. Features only.
data_dictionary.csvMeaning, formula and missing-value handling of every column.

The hidden test input combines the blind-test rows (606 rows). It has the same columns as blind_test_features.csv: observation_id, fiscal_year, industry, industry_description and 31 numeric EDGAR variables and ratios. It has no CIK, ticker, company_name or label column, so your preprocessing must not depend on them. Many numeric values are missing (they are never zero-filled), so impute.

2. The predict.py interface

The evaluator runs exactly this command in an isolated sandbox, with the submission folder as the working directory:

python predict.py \
  --input  /data/test_features.csv \
  --model  /submission/<your model file> \
  --output /output/predictions.csv

Your script must: load the model from --model; read the features from --input; apply the same preprocessing you used in training; write the predictions to --output; and exit with status 0. The --model path is your file named model.* (for example model.joblib), or your only model file if it has another name.

Output format

observation_id,predicted_probability
BT0001,0.1834
BT0002,0.7012
...
  • Exactly one row per input row, with every observation_id from the input, each once. Row order does not matter; rows are matched by id.
  • predicted_probability is the probability that AAER_related = 1: a finite number in [0, 1]. No blanks, NaN or infinity.
  • Header row required. An optional predicted_class column (0/1) is allowed and ignored; any other column is an error.
  • At most 50 MB.

3. Packaging

Upload one ZIP containing predict.py at the top level (a single wrapping folder is fine), your model file and any helper modules or small data files. Or upload predict.py, the model and up to 10 supporting files separately.

my_submission.zip
├── predict.py
├── model.joblib
├── features.py          (optional helper module, imported by predict.py)
└── requirements.txt     (optional, informational; nothing is installed)
  • Max 100 MB compressed, 300 MB extracted, 200 files, 4 folder levels.
  • Allowed types: .py, .joblib, .pkl, .pickle, .json, .txt, .md, .csv, .ubj, .model, .bin, .npy, .npz, .yaml, .yml, .cfg, .ini, .lgb, .xgb, .cbm.
  • Not allowed: symbolic links, hidden files, absolute or .. paths, password-protected ZIPs, compiled extensions (.so), notebooks. __MACOSX, .DS_Store and __pycache__ are dropped automatically.
  • Rejected uploads don't count toward your limit. Accepted uploads always do, even if the evaluation fails, so test locally first.

4. The evaluation environment

Your script runs with Python 3.12 and exactly these packages. Nothing else can be installed, and there is no internet access. Train with the same versions: a model pickled with a different scikit-learn version may fail to load.

numpy==2.3.4
pandas==2.3.3
scipy==1.16.3
scikit-learn==1.7.2
joblib==1.5.2
threadpoolctl==3.6.0
xgboost==3.1.1
lightgbm==4.6.0
CPU / memory2 cores, 4 GB RAM (the process is killed beyond that)
Time120 seconds wall clock, including loading the model
NetworkNone
Files/submission (your files, read-only), /data (test features, read-only), /output (writable, 50 MB), /tmp (scratch, counts against memory)
GPUNone

5. Measure AUPRC yourself

AUPRC (area under the precision–recall curve, also called average precision) asks: if we sort all firm-years from highest to lowest predicted probability, how quickly do the real AAER cases show up at the top? It uses your probabilities, not 0/1 predictions, and needs no threshold.

1.0Perfect: every positive is ranked above every negative.
≈ 0.25No skill. A random model scores the positive rate (about 1 in 4 here), not 0.5 as with AUROC.
0.3 – 0.4Roughly where the example baseline lands on companies it has not seen. Improving on it is real progress.

a) In your training script

The server uses scikit-learn's average_precision_score, so this gives exactly the same number:

from sklearn.metrics import average_precision_score

p_val = model.predict_proba(X_val)[:, 1]          # probability of class 1, NOT model.predict()
print("validation AUPRC:", average_precision_score(y_val, p_val))
print("random baseline :", y_val.mean())          # AUPRC of a model with no skill

b) Cross-validation by company (recommended)

validation.csv is one small split, and its score can be noticeably higher or lower than the leaderboard by luck. Averaging over several splits gives a more reliable estimate. Keep all years of a company in the same fold (GroupKFold on CIK), because the hidden test also contains companies your model has never seen:

import numpy as np, pandas as pd
from sklearn.model_selection import GroupKFold
from sklearn.metrics import average_precision_score

df = pd.concat([pd.read_csv("train.csv"), pd.read_csv("validation.csv")], ignore_index=True)
features = [c for c in df.columns if c not in
            {"observation_id", "CIK", "ticker", "company_name", "industry_description", "AAER_related"}]
X, y, groups = df[features], df["AAER_related"], df["CIK"]

scores = []
for train_idx, test_idx in GroupKFold(n_splits=5).split(X, y, groups):
    model = make_model()                          # your model / pipeline, freshly created each fold
    model.fit(X.iloc[train_idx], y.iloc[train_idx])
    p = model.predict_proba(X.iloc[test_idx])[:, 1]
    scores.append(average_precision_score(y.iloc[test_idx], p))
print(f"AUPRC {np.mean(scores):.3f} ± {np.std(scores):.3f}  (random ≈ {y.mean():.3f})")

Compare models by this mean. A difference smaller than about one standard deviation is mostly noise, on the leaderboard too.

c) Score a predictions file (tests your whole predict.py)

The example package includes evaluate.py, which scores a predictions.csv against any labelled file exactly as the server does:

DATA=/path/to/student_data
python predict.py --input "$DATA/validation.csv" --model model.joblib --output val_predictions.csv
python evaluate.py val_predictions.csv "$DATA/validation.csv"

Only score rows the model was not trained on. The shipped model.joblib was refit on train + validation, so it scores close to 1.0 on validation.csv; that number means nothing. Train on train.csv alone when you want to check on validation.

Common mistakes

  • Passing model.predict(X) (0/1 labels) instead of model.predict_proba(X)[:, 1]. It still runs, but throws away the ranking and lowers AUPRC.
  • Computing auc(recall, precision) from precision_recall_curve. That trapezoid version gives a slightly different, over-optimistic number; use average_precision_score.
  • Evaluating on rows the model was trained on (scores near 1.0), or letting the same company appear in both training and evaluation folds.
  • Choosing a threshold to maximise AUPRC. AUPRC doesn't use one; only the order of your probabilities matters.
The formula

Sort rows by predicted probability, highest first. At each position n, precision Pn is the share of positives among the top n rows and recall Rn is the share of all positives found so far. Then AUPRC = Σn (Rn − Rn−1) · Pn: the average precision measured each time another positive is found.

6. Test locally before you submit

python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt          # from the example package
DATA=/path/to/student_data               # your folder with train.csv, validation.csv, ...
python train.py --train "$DATA/train.csv" --validation "$DATA/validation.csv" --output model.joblib
python predict.py --input "$DATA/blind_test_features.csv" \
                  --model model.joblib --output predictions.csv
python check_submission.py predictions.csv "$DATA/blind_test_features.csv"

Replace /path/to/student_data with wherever you saved the data (on Windows, type the full paths instead of $DATA). --train and --validation can be any labelled CSV files, so you can train on your own split or engineered data. Running python train.py with no arguments uses the small bundled sample_train.csv / sample_validation.csv to test the pipeline; that weak model overwrites model.joblib, so retrain on the full data before you submit.

The example package's check_submission.py applies the same format checks as the server, so a file that passes locally will be accepted (it cannot tell you your hidden score).

7. After you submit

Your dashboard shows the status (Queued → Running → Completed, or Failed / Timed out), the last lines of your script's output for debugging, your public score and rank, and how many submissions you have left in the current window.