aloe-siglip2

What ALOE Does

• Starts from a frozen teacher encoder such as supervised ViT-B/16, DINOv3, or SigLIP2.
• Builds a bias-free B-cos student with six-channel B-cos preprocessing and matching width, depth, and token routing.
• Aligns the student to the teacher on unlabeled image data using global embedding matching and token-level supervision at multiple depths.

The paper reports that this alignment is robust across supervised, self-supervised, and vision-language teachers, and is about 100-1000x more data-efficient than training B-cos models from scratch.

Published Checkpoints

The public ALOE backbone family includes:

The corresponding public ImageNet-1k linear-probe classifiers include the aligned backbone and its trained 1,000-class probe:

Main Metrics From The Paper

All numbers below are top-1 accuracy or GridPG localization scores in percent unless noted otherwise.

ImageNet-1k, Published Model Scale

Across every scale, ALOE holds ImageNet-1k linear-probe and k-NN accuracy within a fraction of a point of the teacher while lifting Grid-PG localization far above the teacher's strongest post-hoc explainer (AttnLRP). Per-checkpoint numbers are in the paper.

Explanations And Feature Geometry

ALOE B-cos attributions (middle) are object-centric and class-specific, while the PCA of the final image representation (bottom) shows the teacher's global feature geometry is preserved — semantics stay aligned while explainability improves.

Linear Evaluation On 10 Datasets, ViT-B/16

The 10-dataset protocol covers ImageNet-1k, Caltech101, Oxford Flowers, Food101, FGVC Aircraft, DTD, Stanford Cars, SUN397, CIFAR-10, and CIFAR-100.

Zero-Shot ImageNet-1k With SigLIP2 Text Encoder

Dense Prediction

For monocular depth with frozen ViT-B/16 features and a linear probe, ALOE improves over B-cosification on NYUv2-style depth metrics:

Usage

These models use custom ALOE Transformers code, so loading requires trust_remote_code=True.

from transformers import AutoImageProcessor, AutoModel

repo_id = "rmaser/aloe-dinov3-base"
processor = AutoImageProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

Hidden States Are Pre-Norm

With output_hidden_states=True, hidden_states[i] is the raw output of block i — in particular hidden_states[-1] is not last_hidden_state, which additionally passes through post_layernorm. This is the convention the distillation loss was defined against, and recent Transformers versions report the post-norm tensor in hidden_states[-1] on the official DINOv3 and ViT models, so the two APIs differ at that one index.

Use hidden_states[i] for anything layer-wise. These models supervise blocks n/3 - 1, 2n/3 - 1 and n - 1 of n, reaching 0.90–0.995 cosine similarity to the teacher's corresponding pre-norm features. Note that t