Toxicity (Tox21 mean ROC-AUC)

Licence to confirm before publication on the hub. Source(s) tox21 declare no known licence (an unknown licence is gated until confirmed in a publication block).

What is measured. Twelve binary in-vitro toxicity read-outs of the 2014 Tox21 Data Challenge (seven nuclear-receptor assays, NR-, and five stress-response assays, SR-), from the SMILES: MoleculeNet Tox21 (Wu et al., Chem. Sci. 2018) as packaged by scikit-fingerprints, 7,831 molecules. Labels are sparse: a molecule was not run in every assay, and a missing label is never read as a negative. scikit-fingerprints removed spurious hydrogens from 8 aluminium SMILES that newer RDKit releases refuse (listed on its card); 1 is in the OGB test part.

Split. The published OGB scaffold split (ogb_splits_tox21.json: 6,264 / 783 / 784), whole Bemis-Murcko scaffolds per part. atlas-full-v1 takes the whole OGB test part (784 molecules) and the whole train part (6,264); atlas-n100-v1 drew 100 test molecules from the test part and 500 train molecules from the train part by seed-hash rank (seed 20261006), unstratified (twelve overlapping sparse labels have no single stratum). The valid part is not used. Scaffold disjointness is re-checked when the subsets are built.

Test size. The leaderboard version is atlas-full-v1 on every profile: the whole OGB test part (784 molecules) and the whole train part (6,264), because a SMILES probe or a small graph network costs seconds per thousand molecules. The earlier atlas-n100-v1 (100 test / 500 train, drawn as described above) stays frozen and its cells stay current; it is a separate ranking, never mixed with the full one.

Metric. For each assay, ROC-AUC over the test molecules where that assay was measured (mask), then the unweighted mean over assays. An assay with fewer than 3 measured positives (or negatives) among the 100 test molecules is excluded from the mean and listed in the result's details. Which assays count is fixed by the frozen subset, so it is the same for every model: in atlas-full-v1 all 12 assays count (22 to 118 measured positives each, 471 to 709 measured molecules); in atlas-n100-v1 NR-ER-LBD has a single measured positive and is excluded, and the other 11 assays (3 to 12 positives each) are averaged.

Probe (mean_roc_auc_probe, mean_average_precision_probe). Frozen embeddings of the SMILES (last layer, mean over tokens); one logistic head per assay (C chosen by stratified CV on log-loss inside train), fit on the train molecules where that assay was measured (4,771 to 5,834 of the 6,264; 385 to 467 of the 500 in atlas-n100-v1).

Supervised (mean_roc_auc_supervised, mean_average_precision_supervised). Multi-task frameworks with no pretrained weights are trained from scratch on the same train molecules, with the label list (NaN = not measured), and score the test molecules for all twelve assays.

Uncertainty. 95 % bootstrap interval over test molecules, 1,000 resamples; within a resample an assay left with a single class drops out of that resample's mean.

Caveats. Per-assay AUCs rest on 22 to 118 positives (a few to about fifteen in atlas-n100-v1), so the mean is what is ranked. The atlas-full-v1 test is the OGB scaffold test part, the one of the literature's scaffold-split numbers, but the train is the OGB train part without its valid part, so values are close to, not equal to, published ones. Many Tox21 compounds are drugs, pesticides and industrial chemicals that are in PubChem, the pretraining corpus of several chemical language models (structures only, not these labels).

Data

• tox21: hf_dataset scikit-fingerprints/MoleculeNet_Tox21 at revision cc0f82565871944a67abb2eadbf73ee6d9664661, licence unknown, catalogued as chem-ml/moleculenet-tox21

Subset

• seed 20261006
• test: 100 item(s)
• train: 500 item(s), disjoint by scaffold
• leaderboard version atlas-full-v1 on every profile: the whole eligible test and train pool (the sizes above define the n version)

Metrics

• mean_roc_auc_probe — Mean ROC-AUC (logistic probe per assay); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• mean_average_precision_probe — Mean average precision (logistic probe per assay); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• mean_roc_auc_supervised — Mean ROC-AUC (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• mean_average_precision_supervised — Mean average precision (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples

Two training molecules of the frozen subset (input: the SMILES; expected output: twelve scores, one per assay, in the order NR-AR, NR-AR-LBD, NR-AhR, NR-Aromatase, NR-ER, NR-ER-LBD, NR-PPAR-gamma, SR-ARE, SR-ATAD5, SR-HSE, SR-MMP, SR-p53; - = not measured, never scored):

item_id	molecule	SMILES	label
tox21-00022	4-aminoazobenzene	Nc1ccc(/N=N/c2ccccc2)cc1	0 0 1 - 1 0 - 1 1 0 - 0
tox21-00098	2,4-hexadienal	C/C=C/C=C/C=O	0 0 0 0 0 0 0 1 0 0 0 0

The azo dye is active in the aryl-hydrocarbon receptor (NR-AhR), oestrogen receptor (NR-ER), antioxidant response (SR-ARE) and genotoxicity (SR-ATAD5) assays; the reactive aldehyde only in the antioxidant response.