How to submit
Everything your submission needs: the data, the predict.py interface, the output format and the evaluation environment.
1. The task
Predict AAER_related: 1 if a company-fiscal-year is linked to an SEC Accounting and Auditing Enforcement Release, 0 for a matched control firm-year. About one row in four is positive, so accuracy is misleading; the competition is scored by AUPRC (average precision) on predicted probabilities.
File (in student_data/) | Use |
|---|---|
train.csv, validation.csv | Labelled firm-years for training and model selection (label column AAER_related). |
blind_test_features.csv | Blind test A — unseen companies. Features only. |
future_blind_test_features.csv | Blind test B — fiscal years 2019–2021. Features only. |
data_dictionary.csv | Meaning, formula and missing-value handling of every column. |
The hidden test input combines the blind-test rows (606 rows). It has the same columns as blind_test_features.csv: observation_id, fiscal_year, industry, industry_description and 31 numeric EDGAR variables and ratios. It has no CIK, ticker, company_name or label column, so your preprocessing must not depend on them. Many numeric values are missing (they are never zero-filled), so impute.
2. The predict.py interface
The evaluator runs exactly this command in an isolated sandbox, with the submission folder as the working directory:
python predict.py \
--input /data/test_features.csv \
--model /submission/<your model file> \
--output /output/predictions.csv
Your script must: load the model from --model; read the features from --input; apply the same preprocessing you used in training; write the predictions to --output; and exit with status 0. The --model path is your file named model.* (for example model.joblib), or your only model file if it has another name.
Output format
observation_id,predicted_probability
BT0001,0.1834
BT0002,0.7012
...
- Exactly one row per input row, with every
observation_idfrom the input, each once. Row order does not matter; rows are matched by id. predicted_probabilityis the probability thatAAER_related = 1: a finite number in [0, 1]. No blanks, NaN or infinity.- Header row required. An optional
predicted_classcolumn (0/1) is allowed and ignored; any other column is an error. - At most 50 MB.
3. Packaging
Upload one ZIP containing predict.py at the top level (a single wrapping folder is fine), your model file and any helper modules or small data files. Or upload predict.py, the model and up to 10 supporting files separately.
my_submission.zip
├── predict.py
├── model.joblib
├── features.py (optional helper module, imported by predict.py)
└── requirements.txt (optional, informational; nothing is installed)
- Max 100 MB compressed, 300 MB extracted, 200 files, 4 folder levels.
- Allowed types:
.py,.joblib,.pkl,.pickle,.json,.txt,.md,.csv,.ubj,.model,.bin,.npy,.npz,.yaml,.yml,.cfg,.ini,.lgb,.xgb,.cbm. - Not allowed: symbolic links, hidden files, absolute or
..paths, password-protected ZIPs, compiled extensions (.so), notebooks.__MACOSX,.DS_Storeand__pycache__are dropped automatically. - Rejected uploads don't count toward your limit. Accepted uploads always do, even if the evaluation fails, so test locally first.
4. The evaluation environment
Your script runs with Python 3.12 and exactly these packages. Nothing else can be installed, and there is no internet access. Train with the same versions: a model pickled with a different scikit-learn version may fail to load.
numpy==2.3.4
pandas==2.3.3
scipy==1.16.3
scikit-learn==1.7.2
joblib==1.5.2
threadpoolctl==3.6.0
xgboost==3.1.1
lightgbm==4.6.0
| CPU / memory | 2 cores, 4 GB RAM (the process is killed beyond that) |
|---|---|
| Time | 120 seconds wall clock, including loading the model |
| Network | None |
| Files | /submission (your files, read-only), /data (test features, read-only), /output (writable, 50 MB), /tmp (scratch, counts against memory) |
| GPU | None |
5. Measure AUPRC yourself
AUPRC (area under the precision–recall curve, also called average precision) asks: if we sort all firm-years from highest to lowest predicted probability, how quickly do the real AAER cases show up at the top? It uses your probabilities, not 0/1 predictions, and needs no threshold.
| 1.0 | Perfect: every positive is ranked above every negative. |
|---|---|
| ≈ 0.25 | No skill. A random model scores the positive rate (about 1 in 4 here), not 0.5 as with AUROC. |
| 0.3 – 0.4 | Roughly where the example baseline lands on companies it has not seen. Improving on it is real progress. |
a) In your training script
The server uses scikit-learn's average_precision_score, so this gives exactly the same number:
from sklearn.metrics import average_precision_score
p_val = model.predict_proba(X_val)[:, 1] # probability of class 1, NOT model.predict()
print("validation AUPRC:", average_precision_score(y_val, p_val))
print("random baseline :", y_val.mean()) # AUPRC of a model with no skill
b) Cross-validation by company (recommended)
validation.csv is one small split, and its score can be noticeably higher or lower than the leaderboard by luck. Averaging over several splits gives a more reliable estimate. Keep all years of a company in the same fold (GroupKFold on CIK), because the hidden test also contains companies your model has never seen:
import numpy as np, pandas as pd
from sklearn.model_selection import GroupKFold
from sklearn.metrics import average_precision_score
df = pd.concat([pd.read_csv("train.csv"), pd.read_csv("validation.csv")], ignore_index=True)
features = [c for c in df.columns if c not in
{"observation_id", "CIK", "ticker", "company_name", "industry_description", "AAER_related"}]
X, y, groups = df[features], df["AAER_related"], df["CIK"]
scores = []
for train_idx, test_idx in GroupKFold(n_splits=5).split(X, y, groups):
model = make_model() # your model / pipeline, freshly created each fold
model.fit(X.iloc[train_idx], y.iloc[train_idx])
p = model.predict_proba(X.iloc[test_idx])[:, 1]
scores.append(average_precision_score(y.iloc[test_idx], p))
print(f"AUPRC {np.mean(scores):.3f} ± {np.std(scores):.3f} (random ≈ {y.mean():.3f})")
Compare models by this mean. A difference smaller than about one standard deviation is mostly noise, on the leaderboard too.
c) Score a predictions file (tests your whole predict.py)
The example package includes evaluate.py, which scores a predictions.csv against any labelled file exactly as the server does:
DATA=/path/to/student_data
python predict.py --input "$DATA/validation.csv" --model model.joblib --output val_predictions.csv
python evaluate.py val_predictions.csv "$DATA/validation.csv"
Only score rows the model was not trained on. The shipped model.joblib was refit on train + validation, so it scores close to 1.0 on validation.csv; that number means nothing. Train on train.csv alone when you want to check on validation.
Common mistakes
- Passing
model.predict(X)(0/1 labels) instead ofmodel.predict_proba(X)[:, 1]. It still runs, but throws away the ranking and lowers AUPRC. - Computing
auc(recall, precision)fromprecision_recall_curve. That trapezoid version gives a slightly different, over-optimistic number; useaverage_precision_score. - Evaluating on rows the model was trained on (scores near 1.0), or letting the same company appear in both training and evaluation folds.
- Choosing a threshold to maximise AUPRC. AUPRC doesn't use one; only the order of your probabilities matters.
The formula
Sort rows by predicted probability, highest first. At each position n, precision Pn is the share of positives among the top n rows and recall Rn is the share of all positives found so far. Then AUPRC = Σn (Rn − Rn−1) · Pn: the average precision measured each time another positive is found.
6. Test locally before you submit
python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt # from the example package
DATA=/path/to/student_data # your folder with train.csv, validation.csv, ...
python train.py --train "$DATA/train.csv" --validation "$DATA/validation.csv" --output model.joblib
python predict.py --input "$DATA/blind_test_features.csv" \
--model model.joblib --output predictions.csv
python check_submission.py predictions.csv "$DATA/blind_test_features.csv"
Replace /path/to/student_data with wherever you saved the data (on Windows, type the full paths instead of $DATA). --train and --validation can be any labelled CSV files, so you can train on your own split or engineered data. Running python train.py with no arguments uses the small bundled sample_train.csv / sample_validation.csv to test the pipeline; that weak model overwrites model.joblib, so retrain on the full data before you submit.
The example package's check_submission.py applies the same format checks as the server, so a file that passes locally will be accepted (it cannot tell you your hidden score).
7. After you submit
Your dashboard shows the status (Queued → Running → Completed, or Failed / Timed out), the last lines of your script's output for debugging, your public score and rank, and how many submissions you have left in the current window.