Atlas
Workbench
Sign in
Models / bio-nlp / biomedclip-pubmedbert-256-vit-base-patch16-224 / benchmarks

bio-nlp/biomedclip-pubmedbert-256-vit-base-patch16-224

Visibility: Public
Model cardFilesBenchmarks
Group by

chest-xray-pneumonia medical-imaging/chest-xray-pneumonia

Pediatric pneumonia detection (chest X-ray) medical-imaging/pediatric-pneumonia-detection

Drawn from: chest-xray-pneumonia

atlas-full-v1

Metric Value 95% CI n Rank Best in catalogue Source Overlap Board
ROC-AUC (zero-shot) ↑ higher is better · zero-shot 0.909 best [0.886, 0.931] 624 1 / 2 0.909 reproduced unknown Board
ROC-AUC (logistic probe) ↑ higher is better · probe 0.964 [0.950, 0.976] 624 3 / 5 0.992 reproduced unknown Board
Accuracy (zero-shot) ↑ higher is better · zero-shot · secondary 0.619 best [0.590, 0.649] 624 — 0.619 reproduced unknown Board
Sensitivity, pneumonia (zero-shot) ↑ higher is better · zero-shot · secondary 0.390 best [0.344, 0.438] 624 — 0.390 reproduced unknown Board
Specificity, pneumonia (zero-shot) ↑ higher is better · zero-shot · secondary 1.000 best [1.000, 1.000] 624 — 1.000 reproduced unknown Board
Average precision (zero-shot) ↑ higher is better · zero-shot · secondary 0.953 best [0.940, 0.964] 624 — 0.953 reproduced unknown Board
Accuracy (logistic probe) ↑ higher is better · probe · secondary 0.864 [0.840, 0.888] 624 — 0.962 reproduced unknown Board
Sensitivity, pneumonia (logistic probe) ↑ higher is better · probe · secondary 0.990 [0.979, 1.000] 624 — 0.997 reproduced unknown Board
Specificity, pneumonia (logistic probe) ↑ higher is better · probe · secondary 0.654 [0.598, 0.718] 624 — 0.902 reproduced unknown Board
Average precision (logistic probe) ↑ higher is better · probe · secondary 0.977 [0.967, 0.986] 624 — 0.994 reproduced unknown Board

atlas-n100-v1

Metric Value 95% CI n Rank Best in catalogue Source Overlap Board
ROC-AUC (zero-shot) ↑ higher is better · zero-shot 0.912 best [0.851, 0.959] 100 1 / 1 0.912 reproduced unknown Board
Accuracy (zero-shot) ↑ higher is better · zero-shot · secondary 0.600 best [0.530, 0.670] 100 — 0.600 reproduced unknown Board
Sensitivity, pneumonia (zero-shot) ↑ higher is better · zero-shot · secondary 0.365 best [0.254, 0.476] 100 — 0.365 reproduced unknown Board
Specificity, pneumonia (zero-shot) ↑ higher is better · zero-shot · secondary 1.000 best [1.000, 1.000] 100 — 1.000 reproduced unknown Board
Average precision (zero-shot) ↑ higher is better · zero-shot · secondary 0.951 best [0.916, 0.977] 100 — 0.951 reproduced unknown Board

NIH-Chest-Xray-14 medical-imaging/nih-chest-xray-14

Chest X-ray findings (NIH ChestX-ray14) medical-imaging/chest-xray-findings-classification

Drawn from: NIH-Chest-Xray-14

atlas-full-v2

Metric Value 95% CI n Rank Best in catalogue Source Overlap Board
Mean ROC-AUC, 5 findings (zero-shot) ↑ higher is better · zero-shot
By finding: 5 levels
findingValue [95% CI]n
Atelectasis0.485 [0.439, 0.536]1013
Cardiomegaly0.687 [0.600, 0.768]1013
Consolidation0.691 [0.634, 0.749]1013
Edema0.780 [0.697, 0.852]1013
Pleural Effusion0.747 [0.712, 0.781]1013

levels share items; n is per level

 0.678 best [0.649, 0.705] 1013 1 / 1 0.678 reproduced unknown Board
Mean ROC-AUC, 5 findings (logistic probe) ↑ higher is better · probe
By finding: 5 levels
findingValue [95% CI]n
Atelectasis0.602 [0.552, 0.655]1013
Cardiomegaly0.742 [0.668, 0.816]1013
Consolidation0.597 [0.533, 0.663]1013
Edema0.746 [0.652, 0.831]1013
Pleural Effusion0.757 [0.723, 0.792]1013

levels share items; n is per level

 0.689 best [0.658, 0.718] 1013 1 / 5 0.689 reproduced unknown Board
Mean average precision, 5 findings (zero-shot) ↑ higher is better · zero-shot · secondary
By finding: 5 levels
findingValue [95% CI]n
Atelectasis0.124 [0.104, 0.155]1013
Cardiomegaly0.075 [0.042, 0.144]1013
Consolidation0.149 [0.107, 0.216]1013
Edema0.105 [0.059, 0.199]1013
Pleural Effusion0.381 [0.318, 0.450]1013

levels share items; n is per level

 0.167 best [0.150, 0.199] 1013 — 0.167 reproduced unknown Board
Mean average precision, 5 findings (logistic probe) ↑ higher is better · probe · secondary
By finding: 5 levels
findingValue [95% CI]n
Atelectasis0.198 [0.158, 0.261]1013
Cardiomegaly0.135 [0.065, 0.263]1013
Consolidation0.113 [0.082, 0.166]1013
Edema0.167 [0.069, 0.307]1013
Pleural Effusion0.403 [0.339, 0.475]1013

levels share items; n is per level

 0.203 best [0.177, 0.248] 1013 — 0.203 reproduced unknown Board

PubMedQA bio-nlp/pubmedqa

PubMedQA accuracy bio-nlp/pubmedqa-accuracy

Drawn from: PubMedQA

atlas-full-v1

Metric Value 95% CI n Rank Best in catalogue Source Overlap Board
Accuracy (logistic probe) ↑ higher is better · probe 0.500 [0.462, 0.540] 500 17 / 17 0.576 reproduced
overlap

Train/test overlap declared by the evaluator: PubMed abstracts

 Board
Macro F1 (logistic probe) ↑ higher is better · probe 0.357 [0.315, 0.399] 500 14 / 17 0.444 reproduced
overlap

Train/test overlap declared by the evaluator: PubMed abstracts

 Board

rxrx3-core cell-imaging/rxrx3-core

Cell Painting knockout retrieval (RxRx3-core) cell-imaging/cell-painting-perturbation-retrieval

Drawn from: rxrx3-core

atlas-n100-v1

Metric Value 95% CI n Rank Best in catalogue Source Overlap Board
Replicate mAP (zero-shot) ↑ higher is better · zero-shot 0.087 [0.079, 0.096] 100 2 / 2 0.101 reproduced none declared Board
Knockout accuracy, 20-way (logistic probe) ↑ higher is better · probe 0.080 [0.030, 0.130] 100 2 / 2 0.120 reproduced none declared Board
Fraction retrieved (zero-shot) ↑ higher is better · zero-shot · secondary 0.000 best [0.000, 0.000] 100 — 0.000 reproduced none declared Board
Knockout macro F1, 20-way (logistic probe) ↑ higher is better · probe · secondary 0.072 [0.032, 0.108] 100 — 0.111 reproduced none declared Board
Knockout top-5 accuracy, 20-way (logistic probe) ↑ higher is better · probe · secondary 0.280 [0.210, 0.350] 100 — 0.350 reproduced none declared Board