Medical exam question answering (MedQA USMLE)

Licence to confirm before publication on the hub. The MedQA mirror declares CC-BY-4.0, but the questions are derived from US Medical Licensing Examination practice material whose own terms are not stated upstream. Local evaluation is fine; results must not be published on the hub before legal confirmation.

What is measured. Medical knowledge and clinical reasoning on questions in the style of the US Medical Licensing Examination (Steps 1, 2 and 3), each with four options and one correct answer.

Data. MedQA (Jin et al., Applied Sciences 2021), English USMLE part, in its common 4-option form (GBaker/MedQA-USMLE-4-options): the official training file (10,178 questions) is the train pool and the official test file (1,273) the test pool. Questions are numbered by line (medqa-test-0000).

Items. 100 test questions by seed-hash rank; 500 train questions, excluding any whose normalised text equals a test question's.

Zero-shot (accuracy_zero_shot; capability score.options). A causal language model reads Question: <question>\nA. <option A>\nB. ...\nD. <option D>\nAnswer: and scores the continuations " A", " B", " C", " D"; the most likely letter is its answer (the multiple-choice format of the common evaluation harnesses, zero-shot, no chain of thought). Small models that have not learnt the format sit near chance (0.25).

Probe (accuracy_probe; capability embed.text.pooled). Each question is paired with each of its options as one text, <question>\nAnswer: <option>, and embedded (last layer, mean over tokens; an encoder truncates a pair longer than its window from the end, which can cut the option: see the length note under Caveats). A logistic head is fit on the 2,000 train pairs (correct = 1) and each test question takes the option whose pair it scores highest. The worker never learns which option is correct.

Uncertainty. 95 % bootstrap interval over test questions, 1,000 resamples (about ±0.09 at 0.5 accuracy).

Caveats. MedQA questions circulate widely online; large language models may have seen them, with their answers, in pretraining or instruction tuning; the training-overlap flag of each result says what the model card declares. Answer letters are not balanced in the test file (A 28 %, B 24 %, C 27 %, D 21 %). Question and longest option together are under 250 words for 99 % of questions (536 at most), so most pairs fit a 512-token encoder. The 100 test questions are answered A 28, B 31, C 24, D 17 times: always answering B scores 0.31.

Data

• medqa: hf_dataset GBaker/MedQA-USMLE-4-options at revision 0fb93dd23a7339b6dcd27e241cb9b5eca62d4d18, licence cc-by-4.0, catalogued as bio-nlp/medqa-usmle-4-options

Subset

• seed 20261006
• test: 100 item(s)
• train: 500 item(s), disjoint by question_key

Metrics

• accuracy_zero_shot — Accuracy (answer-letter likelihood); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• accuracy_probe — Accuracy (pairwise logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples

Two training questions (input: the question and its options; expected output: the letter of the correct option):

item_id	question	options	answer
medqa-train-02958	A 27-year-old man presents to the outpatient clinic with a swollen and painful toe. The pain intensity increased further after he went to a party last night. Which of the following is the drug of choice for the treatment of this patient's condition?	A. Aspirin; B. Rasburicase; C. Indomethacin; D. Allopurinol	C
medqa-train-06974	An investigator is measuring the blood calcium level in a sample of female cross country runners and a control group of sedentary females. If she would like to compare the means of the two groups, which statistical test should she use?	A. t-test; B. Linear regression; C. Chi-square test; D. F-test	A

For the probe, the first question becomes four texts, A 27-year-old man ... condition?\nAnswer: Aspirin through ...\nAnswer: Allopurinol, and the head should score the Indomethacin pair highest.