The problem
A DaT scan images where a radioactive tracer binds to dopamine transporters. In a healthy brain it collects in the striatum as two bright commas. In parkinsonian syndromes the tail of each comma — the putamen — fades first, usually on one side, until the striata look like dots.
The training set was 1,362 scans from many cameras and protocols: spacing from 2.3 to 3.9 mm, fields of view differing 3.6-fold, intensities from 3 to 65,466. Submissions were code, run offline on hidden test scans with a three-hour limit. One rule shaped everything: each test scan had to be scored on its own, with no statistics taken across the test set.
The leaderboard ranked by log loss, with AUROC reported beside it. The two measure different things: AUROC only asks whether abnormal scans are ranked above normal ones, while log loss also charges for how confident each probability is.
Normal
Abnormal
Log loss — the ranking metric
y is 1 for an abnormal scan, p the predicted probability that it is abnormal. It scores confidence as well as correctness: calling an abnormal scan abnormal at 0.99 costs 0.01; calling it normal at 0.99 costs 4.61. One confident mistake outweighs hundreds of confident correct answers, which is why calibration decided places here.
AUROC — reported alongside
Take every pair of one abnormal scan (i) and one normal scan (j) and count how often the abnormal one gets the higher probability; ties count half. It depends only on the ranking, so rescaling the probabilities cannot change it — but it can change the log loss.
The architecture
Each scan is resampled, centred on the striatum and cropped, then seen three ways: as it is, registered to a template of normal scans, and registered to a template of severely abnormal scans. Four pillars read those views — three TabICL models on 116 to 132 hand-built striatal measurements, and a 3D convolutional network trained from scratch on the striatal box.
The pillars are averaged in logit space with weights fixed in advance, shifted once to the test set's prevalence, and floored when a scan looks strongly abnormal. Training ran on 6 cloud GPUs in parallel, and every prediction depends on its own scan only.
| Log loss | AUROC | |
|---|---|---|
| Four-pillar model · cross-validation | 0.2208 | 0.9691 |
| Four-pillar model · without the 3D CNN | 0.2240 | 0.9674 |
| Four-pillar model · floor on random scans (control) | 0.2353 | 0.9633 |
| Four-pillar model · public leaderboard | 0.2459 | 0.9596 |
| Four-pillar model · private leaderboard | 0.2816 | — |
| Best private submission · private leaderboard | 0.2763 | 0.9494 |
Every submission
I scored 20 submissions. The first eight came from an earlier phase — classical models, 3D DenseNets, a hybrid, then TabICL. The next twelve came from a four-week second phase built on top of it.
Every one of those twelve that has a local score beat the TabICL model locally. None of the twelve beat it on the private leaderboard. Over all 19 working submissions, public and private scores still ranked models similarly (Spearman +0.82), but each model lost between 0.034 and 0.068 crossing from one to the other.
Every scored submission: local CV, public and private leaderboard
Log loss, lower is better. The shaded half predates the second phase. The coordinate-bug submission (1.06) is left off the scale.
- public leaderboard
- private leaderboard
- local CV (● site-grouped, ■ stratified)
- local CV fitted in-sample
Show data table
| # | Submission | Local CV | CV kind | Public | Private |
|---|---|---|---|---|---|
| 1 | Classical LightGBM + logistic regression | 0.2853 | stratified | 0.2708 | 0.3085 |
| 2 | 3D DenseNet121, one model + temperature | 0.2745 | stratified | 0.3063 | 0.3503 |
| 3 | 3D DenseNet121, five seeds | 0.2649 | stratified | 0.2884 | 0.3356 |
| 4 | Hybrid: deep + classical | 0.2207 | stratified | 0.2601 | 0.2968 |
| 5 | Classical v2 | — | — | 0.2581 | 0.2924 |
| 6 | Classical v3, atlas features | — | — | 0.2630 | 0.3020 |
| 7 | Classical T5, LightGBM + logistic regression | 0.2677 | grouped | 0.2507 | 0.2878 |
| 8 | TabICL on 96 features | 0.2554 | grouped | 0.2424 | 0.2763 |
| 9 | TabICL on 116 features | 0.2422 | site-disjoint | 0.2425 | 0.2870 |
| 10 | Brain-based denominator, both arms | 0.2363 | site-disjoint | 0.2391 | 0.2824 |
| 11 | Plus worse-side mirror | 0.2334 | site-disjoint | 0.2396 | 0.2834 |
| 12 | Rank-mean of nine legs | 0.2349 | site-disjoint | 0.2286 | 0.2810 |
| 13 | Eleven legs, coordinate bug | — | — | 1.0619 | 1.0666 |
| 14 | Table legs + 3D CNN | 0.2257 | site-disjoint | 0.2361 | 0.2854 |
| 15 | Eleven legs incl. 3D CNN + TTA | 0.2260 | site-disjoint | 0.2409 | 0.2856 |
| 16 | Many-leg ensemble, fitted weights | 0.2179 | in-sample | 0.2428 | 0.3011 |
| 17 | 21 fitted parameters | 0.2090 | in-sample | 0.2466 | 0.3145 |
| 18 | Compound blend 30/70 | 0.2260 | site-disjoint | 0.2453 | 0.2858 |
| 19 | Four pillars incl. 3D CNN + risk floor | 0.2208 | site-disjoint | 0.2459 | 0.2816 |
| 20 | Last upload, unidentified | — | — | 0.2389 | 0.2840 |
| Submission | Local CV | Public | Private |
|---|---|---|---|
| TabICL, 96 features | 0.2554 | 0.2424 | 0.2763 |
| TabICL, 116 features | 0.2422 | 0.2425 | 0.2870 |
| Brain-based denominator | 0.2363 | 0.2391 | 0.2824 |
| Rank-mean of nine legs (rule breach) | 0.2349 | 0.2286 | 0.2810 |
| Table legs + 3D CNN | 0.2257 | 0.2361 | 0.2854 |
| Four pillars + 3D CNN + floor | 0.2208 | 0.2459 | 0.2816 |
| Many-leg ensemble | 0.2179 in-sample | 0.2428 | 0.3011 |
| 21 fitted parameters | 0.2090 in-sample | 0.2466 | 0.3145 |
What the second phase tried
Four weeks and 363 logged experiments: better reference regions, mirroring the weaker side, a from-scratch 3D CNN as an extra model, symmetry averaging at test time, many-leg ensembles, risk floors and temperature fits. Several gave real local gains. The chart sets those local gains beside what the public board returned.
One entry deserves its own line. My best public score, 0.2286, came from ranking each scan against the other test scans — which the rules forbid. A legal reconstruction later measured that step's gain at +0.0002 AUROC: noise that happened to land well.
Offline wins, board results
Change in log loss against the model each change was built on. Negative is better. The last offline row is a control: ten label-free noise arms.
- better
- worse
Show data table
| Change | Measured on | Δ log loss |
|---|---|---|
| Brain-based denominator | offline | -0.0059 |
| 3D CNN added to the table model | offline | -0.0040 |
| Worse-side mirror | offline | -0.0029 |
| Three-way equal mean | offline | -0.0022 |
| Ten noise arms, weights fitted in-sample | offline | -0.0077 |
| Brain-based denominator vs base | public | -0.0033 |
| Four equal pillars vs brain-based denominator | public | -0.0012 |
| Worse-side mirror | public | +0.0005 |
| Risk floor on four pillars | public | +0.0048 |
| Four pillars + 3D CNN | public | +0.0068 |
| Table + 3D CNN vs rank-mean | public | +0.0075 |
Why local wins did not reach the leaderboard
The honest answer is that I spent most of the second phase measuring rather than building. I pre-registered experiments, built nested scorers, audited folds and chased the gap between local and leaderboard scores. That discipline caught real errors — but it also meant I kept refining one family of models instead of trying genuinely different ones.
The mistakes, in order of cost. I tuned ensemble weights on the same predictions I scored, which manufactures gains out of noise. My early folds let acquisition sites leak between training and validation, so some ideas were closed on flawed evidence. I optimised a pooled metric that ranked submissions in the wrong order. And the code I shipped did not always compute exactly what I had measured offline.
Two teams that finished ahead of me on the private board published their solutions, and both built different representations early — 2D projections of the striatum, synthetic lesions, deliberately softer calibration — and let the leaderboard test them. Next time I would build first, verify what survives, and keep a simple baseline protected throughout.
A better local score did not buy a better leaderboard
Second-phase submissions with a site-disjoint local CV. Better local CV is to the right. Hollow diamonds had weights fitted on the scored predictions.
Show data table
| Submission | Local CV | Public | Private |
|---|---|---|---|
| TabICL on 116 features | 0.2422 | 0.2425 | 0.2870 |
| Brain-based denominator, both arms | 0.2363 | 0.2391 | 0.2824 |
| Plus worse-side mirror | 0.2334 | 0.2396 | 0.2834 |
| Rank-mean of nine legs | 0.2349 | 0.2286 | 0.2810 |
| Table legs + 3D CNN | 0.2257 | 0.2361 | 0.2854 |
| Eleven legs incl. 3D CNN + TTA | 0.2260 | 0.2409 | 0.2856 |
| Many-leg ensemble, fitted weights | 0.2179 (in-sample) | 0.2428 | 0.3011 |
| 21 fitted parameters | 0.2090 (in-sample) | 0.2466 | 0.3145 |
| Compound blend 30/70 | 0.2260 | 0.2453 | 0.2858 |
| Four pillars incl. 3D CNN + risk floor | 0.2208 | 0.2459 | 0.2816 |
Public vs private: every model lost 0.034 to 0.068
Each point is one submission. The dashed line is private = public + 0.035.
- second phase
- first phase
- best private
Show data table
| Submission | Public | Private | Gap |
|---|---|---|---|
| Classical LightGBM + logistic regression | 0.2708 | 0.3085 | +0.0377 |
| 3D DenseNet121, one model + temperature | 0.3063 | 0.3503 | +0.0440 |
| 3D DenseNet121, five seeds | 0.2884 | 0.3356 | +0.0472 |
| Hybrid: deep + classical | 0.2601 | 0.2968 | +0.0367 |
| Classical v2 | 0.2581 | 0.2924 | +0.0343 |
| Classical v3, atlas features | 0.2630 | 0.3020 | +0.0390 |
| Classical T5, LightGBM + logistic regression | 0.2507 | 0.2878 | +0.0371 |
| TabICL on 96 features | 0.2424 | 0.2763 | +0.0339 |
| TabICL on 116 features | 0.2425 | 0.2870 | +0.0445 |
| Brain-based denominator, both arms | 0.2391 | 0.2824 | +0.0433 |
| Plus worse-side mirror | 0.2396 | 0.2834 | +0.0438 |
| Rank-mean of nine legs | 0.2286 | 0.2810 | +0.0524 |
| Table legs + 3D CNN | 0.2361 | 0.2854 | +0.0493 |
| Eleven legs incl. 3D CNN + TTA | 0.2409 | 0.2856 | +0.0447 |
| Many-leg ensemble, fitted weights | 0.2428 | 0.3011 | +0.0583 |
| 21 fitted parameters | 0.2466 | 0.3145 | +0.0679 |
| Compound blend 30/70 | 0.2453 | 0.2858 | +0.0405 |
| Four pillars incl. 3D CNN + risk floor | 0.2459 | 0.2816 | +0.0357 |
| Last upload, unidentified | 0.2389 | 0.2840 | +0.0451 |
The shake-up
The private set was harder for everyone. The public leader finished 28th, the tenth-placed team 38th, the twelfth 2nd. I went from 4th to 11th.
The private leaderboard reshuffled the public top twelve
- me (ghost_sas), and two teams ahead of me in public
Show data table
| Team | Public rank | Private rank |
|---|---|---|
| Marc-Dvci | 1 | 28 |
| TheAvengers | 2 | 1 |
| South-Wing | 3 | 17 |
| ghost_sas (me) | 4 | 11 |
| Shatatarka | 5 | 7 |
| Shivom | 6 | 3 |
| paulonium | 7 | 6 |
| Tensla | 8 | 9 |
| AAF Team | 9 | 8 |
| venkt | 10 | 38 |
| Suyash92 | 11 | 4 |
| Tigertech | 12 | 2 |
What I would do differently
My biggest mistake was treating the public leaderboard as the truth. Every idea went through the same loop: build it, submit it, read the public score, and drop it if the number went the wrong way. The public set was small and easier than the private one, so that loop was steering me by noise.
The four-pillar model is the clearest example. On the public board it looked like a step backwards, so I moved on. On the private board it turned out to be one of my strongest second-phase submissions. Other ideas were abandoned the same way, on a single public reading, without ever finding out what they were really worth.
Next time the leaderboard becomes one signal among several, not the goal. I would judge a model by how it was built and how it holds up on honest validation — site-disjoint folds, a protected simple baseline, the shipped code checked against what was measured — and use the public score to catch broken submissions, not to decide which ideas live.