dino-with-registers

About

What dino-with-registers is

Vision Transformer (base-sized model) trained using DINOv2, with registers Vision Transformer (ViT) model introduced in the paper Vision Transformers Need Registers by Darcet et al. and first released in this repository. Disclaimer: The team releasing DINOv2 with registers did not write a model card for this model so this model card has been written by the Hugging Face team. Model description The Vision Transformer (ViT) is a transformer encoder model (BERT-like) originally introduced to do supervised image classification on ImageNet. Next, people figured out ways to make ViT work really well on self-supervised image feature extraction (i.e. learning meaningful features, also called embeddings) on images without requiring any labels. Some example papers here include DINOv2 and MAE. The authors of DINOv2 noticed that ViTs have artifacts in attention maps. It’s due to the model using some image patches as “registers”. The authors propose a fix: just add some new tokens (called "register" tokens), which you only use during pre-training (and throw away afterwards). This results in: - no artifacts - interpretable attention maps - and improved performances. alt="drawing" width="600"/ Note that this model does not include any fine-tuned heads. By pre-training the model, it learns an inner representation of images that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled images for instance, you can train a standard classifier by placing a linear layer on top of the pre-trained encoder. One typically places a linear layer on top of the [CLS] token, as the last hidden state of this token can be seen as a representation of an entire image. Intended uses & limitations You can use the raw model for feature extraction. See the model hub to look for fine-tuned versions on a task that interests you. How to use Here is how to use this model: BibTeX entry and citation info

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How embedding models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys dino-with-registers for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (dino-with-registers below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/embeddings \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"dino-with-registers","input":"text to embed"}'

Details

Languages, data & research

Papers

Licence

Open weights

Sources

Weights & code

Explore

More embedding models