MoleculeNet HIV ROC-AUC

Licence to confirm before publication on the hub. Source(s) hiv declare no known licence (an unknown licence is gated until confirmed in a publication block).

What is measured. Whether a small molecule inhibits HIV replication (1: confirmed active or moderately active in the NCI DTP AIDS antiviral screen; 0: inactive), from its SMILES, on MoleculeNet HIV (Wu et al., Chem. Sci. 2018) as packaged by scikit-fingerprints: 41,127 molecules, of which 1,443 (3.5 %) are active. scikit-fingerprints repaired 7 SMILES that newer RDKit releases refuse (hypervalent Al/B/Ge/Cu complexes, listed on its card); 2 of them are in the OGB test part.

Split. The published OGB scaffold split (ogb_splits_hiv.json: 32,901 / 4,113 / 4,113): whole Bemis-Murcko scaffolds per part, the largest scaffold groups to train, the rarest to test. The test pool is the OGB test part (130 actives), the train pool the OGB train part (1,232 actives); the valid part is not used. Scaffold disjointness of the two subsets is re-checked when they are built, not taken from the file.

Enrichment (deviation from the natural rate). At 3.5 % actives, 100 random test molecules would hold 3 or 4 actives and a ROC-AUC with a confidence interval spanning most of [0, 1]. So, in each pool, every active is kept and the inactives are thinned by seed-hash rank (seed 20261006, the ranking the subset builder uses) to four per active; the enriched pools hold 130 actives / 520 inactives (test, 650 molecules) and 1,232 / 4,928 (train, 6,160), which atlas-full-v1 takes whole; the label-stratified draw of atlas-n100-v1 held exactly 20 / 80 in its 100 test molecules and 100 / 400 in its 500 train molecules. ROC-AUC does not depend on the class ratio (in expectation); average precision does, and its chance level here is 0.2. Both are comparable across models of this benchmark, not with papers that use the full 4,113-molecule test part.

Test size. The leaderboard version is atlas-full-v1 on every profile: the whole enriched test pool (650 molecules) and the whole enriched train pool (6,160), because a SMILES probe or a small graph network costs seconds per thousand molecules. The earlier atlas-n100-v1 (100 test / 500 train, drawn as described above) stays frozen and its cells stay current; it is a separate ranking, never mixed with the full one.

Probe (roc_auc_probe, average_precision_probe). Frozen embeddings of the SMILES (last layer, mean over tokens); a logistic head (C chosen by stratified 5-fold CV on log-loss inside train) is fit on the train molecules and scores the test molecules.

Supervised (roc_auc_supervised, average_precision_supervised). Frameworks with no pretrained weights (Chemprop, graph networks) are trained from scratch on the same train molecules and score the same test molecules; kept apart from the probe so "trained on this benchmark" is never ranked against "frozen features".

Uncertainty. 95 % bootstrap interval over test molecules, 1,000 resamples stratified by label.

Caveats. Chemical language models pretrained on PubChem or ZINC have seen most of these structures without their labels; that is not a label leak, but it is an advantage scaffold splitting does not remove. The "moderately active" class is merged with "confirmed active", as in MoleculeNet.

Data

• hiv: hf_dataset scikit-fingerprints/MoleculeNet_HIV at revision 2d353911b426c7fc38a147cbbc116808dd0d4764, licence unknown, catalogued as chem-ml/moleculenet-hiv

Subset

• seed 20261006
• test: 100 item(s), stratified by label
• train: 500 item(s), stratified by label, disjoint by scaffold
• leaderboard version atlas-full-v1 on every profile: the whole eligible test and train pool (the sizes above define the n version)

Metrics

• roc_auc_probe — ROC-AUC (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• average_precision_probe — Average precision (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• roc_auc_supervised — ROC-AUC (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• average_precision_supervised — Average precision (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label

Three training molecules of the frozen subset (input: the SMILES; expected output: a score that ranks HIV-replication inhibitors above the others):

item_id	SMILES	label
hiv-00404	C=Cc1ccc(CC)cn1 (5-ethyl-2-vinylpyridine)	0
hiv-03267	N#Cc1c(N)[nH]c(S)c1C#N	1
hiv-06319	NC(=O)NN=C1CSSC1	1

label 1 merges the screen's "confirmed active" and "moderately active" classes; the item id is the row of the pinned hiv.csv. The leaderboard (atlas-full-v1) scores the whole enriched test pool, 650 molecules (100 in atlas-n100-v1).