BiomedCLIP-PubMedBERT_256-vit_base_patch16_224

About

What BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 is

BiomedCLIP-PubMedBERT256-vitbasepatch16224 BiomedCLIP is a biomedical vision-language foundation model that is pretrained on PMC-15M, a dataset of 15 million figure-caption pairs extracted from biomedical research articles in PubMed Central, using contrastive learning. It uses PubMedBERT as the text encoder and Vision Transformer as the image encoder, with domain-specific adaptations. It can perform various vision-language processing (VLP) tasks such as cross-modal retrieval, image classification, and visual question answering. BiomedCLIP establishes new state of the art in a wide range of standard datasets, and substantially outperforms prior VLP approaches: Contents - Training Data - Model Use - Reference - Limitations - Further Information Training Data We have released BiomedCLIP Data Pipeline at https://github.com/microsoft/BiomedCLIPdatapipeline, which automatically downloads and processes a set of articles from the PubMed Central Open Access dataset. BiomedCLIP builds upon the PMC-15M dataset, which is a large-scale parallel image-text dataset generated by this data pipeline for biomedical vision-language processing. It contains 15 million figure-caption pairs extracted from biomedical research articles in PubMed Central and covers a diverse range of biomedical image types, such as microscopy, radiography, histology, and more. Model Use 1. Environment 2.1 Load from HF hub 2.2 Load from local files Use in Jupyter Notebook Please refer to this example notebook. Intended Use This model is intended to be used solely for (I) future research on visual-language processing and (II) reproducibility of the experimental results reported in the reference paper. Primary Intended Use The primary intended use is to support AI researchers building on top of this work. BiomedCLIP and its associated models should be helpful for exploring various biomedical VLP research questions, especially in the radiology domain. Out-of-Scope Use Any deployed use case of the model --- commercial or otherwise --- is currently out of scope. Although we evaluated the models using a broad set of publicly-available research benchmarks, the models and evaluations are not intended for deployed

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How embedding models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys biomedclip-pubmedbert-256-vit-base-patch16-224 for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (biomedclip-pubmedbert-256-vit-base-patch16-224 below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/embeddings \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"biomedclip-pubmedbert-256-vit-base-patch16-224","input":"text to embed"}'

Details

Languages, data & research

Licence

Open weights

Sources

Weights & code

Explore

More embedding models