Open-PMC: Large-scale medical vision-language pretraining from PubMed Central

Abstract

Better image-text alignment, not just more data.

Biomedical figures are usually compound: a single figure packs several panels, each a different imaging modality, anatomy, or concept, all sharing one caption. Pairing the whole compound figure with that caption injects noise into contrastive pretraining. Open-PMC takes a quality-first approach: decompose compound figures into panels, then align each panel with its caption.

Open-PMC-18M scales this to 18 million subfigure-caption pairs spanning radiology (CT, MRI, X-ray), microscopy (histopathology, blood & tissue), and visible-light photography (dermatology, retinal fundus), using a transformer-based subfigure extractor trained on 500K synthetic compound figures.

Models trained on Open-PMC-18M set a new state of the art in zero-shot medical cross-modal retrieval and improve zero-shot classification and robustness over PMC-15M and BIOMEDICA baselines; dataset, models, and code are all open.

How the dataset is built

Cutting compound figures into panels that actually match the caption

A typical PubMed Central figure is a compound panel: a CT scan beside a histology slide beside a chart, all sharing one caption. Pairing that whole figure with the caption is the misalignment most medical datasets bake in. Open-PMC-18M removes it by decomposing figures into their panels first, then pairing and filtering.

Concretely, we start from 6M PMC compound figures (PMC-6M), run a transformer detector that cuts them into roughly 32M single panels, and keep the 18M panels a relevance classifier judges clinically meaningful, each now aligned to the caption text that describes it.

The full Open-PMC data pipeline: download and parse figure-caption pairs from PubMed Central, decompose compound figures into subfigures, pair and filter for clinical relevance, then contrastively pretrain vision and text encoders. — **The full Open-PMC pipeline.** From open-access PubMed Central articles to trained vision-language encoders: parse figure-caption pairs, decompose compound figures into panels, pair and filter for clinical relevance, then contrastively pretrain.

A worked example: one compound figure, end to end

Source compound figure + captionWe start from a real multi-panel figure and the single caption written around it, drawn from open-access PMC.

PMC-6M · BIOMEDICA

A multi-panel biomedical compound figure with radiology, microscopy, and photography panels

Caption (shared, illustrative). (A) Axial CT of the abdomen. (B) H&E-stained histology of the resected mass. (C) Dermoscopic view of the associated skin lesion.

One caption describes a CT panel, a histology panel, and a dermoscopy panel. Pairing the whole figure with this caption forces the model to align three unrelated images with one text, which is the noise Open-PMC removes.

Subfigure extractionThe DAB-DETR detector draws one bounding box around each panel and crops it out of the compound figure.

DAB-DETR · 98.6% mAP

Running the detector on the figure above returns three bounding boxes, one per panel. Each dashed box below is a cropped-out panel, tagged with the modality it belongs to:

Box ARadiologyCT scan

Box BMicroscopyH&E histology

Box CVisible lightdermoscopy

Subfigure-caption pairingEach detected panel is paired with the source caption, turning one noisy pair into several clean, aligned ones.

1 figure → N pairs

ACT panel: "Axial CT of the abdomen."

BHistology panel: "H&E-stained histology of the resected mass."

CDermoscopy panel: "Dermoscopic view of the associated skin lesion."

Clinical-relevance filteringMetadata labels plus a ResNet-101 relevance classifier drop non-clinical panels, yielding the final 18M pairs.

ResNet-101 filter

~32Mraw panels

→

26Mclinical / microscopy

→

18Mhigh-quality pairs

What's inside Open-PMC-18M

Open-PMC-18M dataset statistics: distribution of imaging modalities, number of subfigures per compound figure, and caption-length distribution. — **Dataset composition.** Modality distribution, subfigures per compound figure, and caption lengths in Open-PMC-18M.

The final dataset is dominated by pathology and microscopy, with radiology and visible-light photography making up the rest, and captions that are often long and detailed.

Pathology & microscopy 73%
Radiology 18%
Visible-light photography 8%

Captions average 165.8 tokens; 19.5% exceed 256 tokens, up to a maximum of 7,352.

How the detector works: DAB-DETR

Step 2 above is the technical core. To train a panel detector at scale we need labeled compound figures, but annotated ones are scarce. So we compose synthetic compound figures from single-panel images, which hands us exact ground-truth boxes for free, then train on 500K of them.

Randomized grid layouts, from standard m×n to custom arrangements
Varied panel margins and aspect ratios
Label schemes 1, a, a1, a-1, placed inside or outside panels
Exact ground-truth boxes recorded for every panel

Synthetic compound-figure generation pipeline: a Sampler selects single-panel images and layouts, a Labeler assigns panel labels, producing synthetic compound figures with known ground-truth boxes to train DAB-DETR. — **Building the training data.** A *Sampler* draws single-panel images and a layout, a *Labeler* adds panel labels (1, a, a1, a-1), and the panels are composed into a synthetic compound figure with exact ground-truth boxes.

1Panel separation as object detection

Classical splitters rely on whitespace, edges, or layout heuristics and break on irregular panels. Instead we frame extraction as object detection: given a compound figure, predict a bounding box around every subpanel. We use DAB-DETR (Dynamic Anchor Box DETR), a transformer detector whose queries are explicit 4-D anchor boxes (x, y, w, h) refined layer by layer. Anchoring queries to real boxes gives sharper localization and faster convergence than vanilla DETR, which matters for the tight, non-uniform layouts of published figures.

2Trained on 500K synthetic figures

Annotated compound figures are scarce; prior work trained on just 2,069 hand-labeled MedICaT figures. We reverse the problem: instead of decomposing figures, we compose them. A generator samples single-panel images, arranges them on randomized grids with varied margins, label schemes (1, a, a1, a-1), and aspect ratios, and records exact ground-truth boxes for free. Trained on 500,000 such figures, the detector hits 98.6% mAP on held-out synthetic data and beats the MedICaT model on real ImageCLEF 2016 figures (36.9 vs 28.2 mAP).

ROCOSICAPHAM10000MedMNISTPAD-UFES-20PlotQA

Qualitative subfigure detection: DAB-DETR localizes and separates panels in real PMC compound figures and ImageCLEF 2016 examples, each panel enclosed in a colored bounding box. — **Qualitative detection results.** The trained detector localizes and separates panels in real PMC compound figures (left) and ImageCLEF 2016 examples (right), even with heterogeneous panels and non-uniform layouts.

Results

State-of-the-art zero-shot retrieval

Zero-shot cross-modal retrieval with the released Open-PMC-18M checkpoint, with no task-specific fine-tuning. Recall@200 for image→text and text→image across three modalities.

Zero-shot cross-modal retrieval · Recall@200

Recall@200 for the released Open-PMC-18M checkpoint, no task-specific fine-tuning. Higher is better.
Benchmark	Image → Text	Text → Image
MIMIC-IV-CXR Chest X-ray · test	27.47%	28.14%
Quilt-1M Histopathology · val	25.53%	27.16%
DeepEyeNet Retina · test	19.30%	20.48%

Produced by the scripts in evaluation/zero_shot_retrieval/; by default they pull the released Open-PMC-18M checkpoint from the Hugging Face Hub, so you just point each script at its dataset root directory.

Subfigure detection vs. prior work

Subfigure separation performance. Training on 500K synthetic compound figures far exceeds the prior detector trained on the 2,069-figure MedICaT set.
Model	Synthetic mAP	Synthetic F1	ImageCLEF 2016 mAP	ImageCLEF 2016 F1
Previous model (MedICaT)	33.22	73.18	28.20	64.85
Our model (DAB-DETR)	98.58	99.96	36.88	73.55

Across zero-shot classification (18 tasks in radiology, microscopy, and VLP), Open-PMC-18M also achieves the highest average performance and improved robustness under visual perturbations; see the paper for full tables.

Get started

Load the checkpoint and run retrieval

load_checkpoint.py

# Load the Open-PMC-18M CLIP checkpoint via open_clip
import torch, open_clip
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms(
    "hf-hub:vector-institute/open-pmc-18m-clip"
)
tokenizer = open_clip.get_tokenizer(
    "hf-hub:vector-institute/open-pmc-18m-clip"
)
model.eval()

# Encode an image and candidate captions
image = preprocess(Image.open("panel.png")).unsqueeze(0)
text  = tokenizer(["chest x-ray", "H&E histology", "retinal fundus"])

with torch.no_grad():
    img_feat = model.encode_image(image)
    txt_feat = model.encode_text(text)
    img_feat /= img_feat.norm(dim=-1, keepdim=True)
    txt_feat /= txt_feat.norm(dim=-1, keepdim=True)
    probs = (100.0 * img_feat @ txt_feat.T).softmax(dim=-1)
print(probs)

zero_shot_retrieval.sh

# Reproduce the retrieval numbers above.
# Scripts default to the released HF checkpoint;
# just set each dataset's root directory.

# Histopathology: Quilt-1M
QUILT_ROOT_DIR=/data/quilt \
  bash evaluation/zero_shot_retrieval/quilt.sh

# Chest X-ray: MIMIC-IV-CXR
MIMIC_ROOT_DIR=/data/mimic \
  bash evaluation/zero_shot_retrieval/mimic.sh

# Retina: DeepEyeNet
DEN_ROOT_DIR=/data/deepeyenet \
  bash evaluation/zero_shot_retrieval/deepeyenet.sh

# To score a local checkpoint instead:
# export CKPT=/path/to/open_clip_model.pt

See evaluation/README.md for the full list of datasets and options, including zero-shot classification.

Citation

Cite Open-PMC

@article{baghbanzadeh2025open,
  title={Open-pmc-18m: A high-fidelity large scale medical dataset
         for multimodal representation learning},
  author={Baghbanzadeh, Negin and Islam, Mohammed Saidul and
          Ashkezari, Sajad and Dolatabadi, Elham and Afkanpour, Arash},
  journal={arXiv preprint arXiv:2506.02738},
  year={2025}
}

@article{baghbanzadeh2025advancing,
  title   = {Advancing Medical Representation Learning Through
             High-Quality Data},
  author  = {Baghbanzadeh, Negin and Fallahpour, Adibvafa and
             Parhizkar, Yasaman and Ogidi, Franklin and Roy, Shuvendu and
             Ashkezari, Sajad and Khazaie, Vahid Reza and Colacci, Michael
             and Etemad, Ali and Afkanpour, Arash and Dolatabadi, Elham},
  journal = {arXiv preprint arXiv:2503.14377},
  year    = {2025}
}

Dataset & code: github.com/VectorInstitute/pmc-data-extraction · 🤗 Open-PMC-18M collection