phikon

Phikon-v2 is a Vision Transformer Large pre-trained with Dinov2 self-supervised method on PANCAN-XL, a dataset of 450M 20x magnification histology images sampled from 60K whole slide images. PANCAN-XL only incorporates publicly available datasets: CPTAC (6,193 WSI) and TCGA (29,502 WSI) for malignant tissue, and GTEx for normal tissue (13,302 WSI).

Phikon-v2 improves upon Phikon, our previous foundation model pre-trained with iBOT on 40M histology images from TCGA (6k WSI), on a large variety of weakly-supervised tasks tailored for biomarker discovery. Phikon-v2 is evaluated on external cohorts to avoid any data contamination with PANCAN-XL pre-training dataset, and benchmarked against an exhaustive panel of representation learning and foundation models.

Model Description

• Developed by: Owkin, Inc
• Model type: Pretrained vision backbone (ViT-L/16 via DINOv2)
• Pretraining dataset: PANCAN-XL, sourced from public histology collections (TCGA, CPTAC, GTEx, TCIA and others).
• Paper: Arxiv
• License: Owkin non-commercical licence

How To Use (Feature Extraction)

The following code snippet allows you to extract features from histology images using Phikon-v2 (CLS token). These features can then be used for downstream applications such as ROI classification (via linear or knn probing), slide classification (via multiple instance learning), segmentation (via ViT-Adapter for instance), etc.

from PIL import Image
import torch
from transformers import AutoImageProcessor, AutoModel

# Load an image
image = Image.open(
    requests.get(
        "https://github.com/owkin/HistoSSLscaling/blob/main/assets/example.tif?raw=true",
        stream=True
    ).raw
)

# Load phikon-v2
processor = AutoImageProcessor.from_pretrained("owkin/phikon-v2")
model = AutoModel.from_pretrained("owkin/phikon-v2")
model.eval()

# Process the image
inputs = processor(image, return_tensors="pt")

# Get the features
with torch.inference_mode():
    outputs = model(**inputs)
    features = outputs.last_hidden_state[:, 0, :]  # (1, 1024) shape

assert features.shape == (1, 1024)

Direct Use (with Pre-Extracted and Frozen Features)

Phikon-v2 can be used with or without fine-tuning on different downstream applications, on top of which slide-classification using multiple instance learning algorithms (such as ABMIL).

Downstream Use (Finetuning)

You can fine-tune the model on tile-level downstream tasks.

Training Details

• Training data: PANCAN-XL, a pretraining dataset composed of 456,060,584 [224×224] histology images at 20× resolution, sampled from 60k H&E WSIs.
• Training regime: fp16 using PyTorch-FSDP mixed-precision.
• Training objective: DINOv2 SSL recipe with the following losses:
  • DINO self-distillation loss with multi-crop
  • iBOT masked-image modeling loss
  • KoLeo regularization on [CLS] tokens
• Training length: 100,000 iterations with a batch size of 4,096
• Model architecture: ViT-Large (0.3B params): Patch size 16, embedding dimension 1024, 16 heads, MLP FFN
• Hardware used: 32x4 Nvidia V100 32GB
• Hours trained: Approx 4,300 GPU hours (33 hours total)
• Platform: French supercluster Jean-Zay

Software Dependencies

Python Packages

• torch>==2.0.0: https://pytorch.org
• torchvision>=0.15.0: https://pytorch.org/vision/stable/index.html
• xformers>=0.0.18: https://github.com/facebookresearch/xformers

Repositories

• DINOv2 (self-supervised learning): https://github.com/facebookresearch/dinov2

Contact

For any additional questions or comments, contact Alexandre Filiot (alexandre.filiot@owkin.com).

How to cite

@misc{filiot2024phikonv2largepublicfeature,
      title={Phikon-v2, A large and public feature extractor for biomarker prediction},
      author={Alexandre Filiot and Paul Jacob and Alice Mac Kain and Charlie Saillard},
      year={2024},
      eprint={2409.09173},
      archivePrefix={arXiv},
      primaryClass={eess.IV},
      url={https://arxiv.org/abs/2409.09173},
}

Acknowledgements

We thank DINOv2 authors for the amazing contribution [1].

Computing resources

This work was granted access to the HPC resources of IDRIS under the allocation 2023-A0141012519 made by GENCI.

Datasets

The results published here are partly based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga. The Genotype-Tissue Expression (GTEx) Project was supported by the Common Fund of the Office of the Director of the National Institutes of Health, and by NCI, NHGRI, NHLBI, NIDA, NIMH, and NINDS. The data used for the analyses described in this manuscript were obtained from the GTEx Portal on 07/01/2023.

Third-party licenses

Vision Transformers architectures were derived from facebookresearch/dino (Apache License 2.0), huggingface/pytorch-image-models (Apache License 2.0). This code is built upon DINOv2 repository (Apache License 2.0).

The following table provides the license associated with each datasets used for pre-training Phikon-v2.