Mage-V

About

What Mage-V is

Mage-ViT is the visual encoder at the core of Mage-VL. It is a Codec-ViT built primarily for video, where a single image is simply the degenerate one-frame case. Mage-ViT follows a codec-aligned sparsity principle: visual tokens should be spent where a video codec spends bits, since codec bit-allocation is a natural proxy for spatio-temporal importance. On a shared 16×16 patch grid it keeps every anchor (I-frame) patch and only the motion-salient predicted (P-frame) patches, while a shared 3D rotary position encoding preserves spatio-temporal structure even after large fractions of the grid are dropped. This repository is the ViT-pre-trained checkpoint only — it has not gone through the joint VLM training with the language model. Use it as a data-efficient, codec-native visual encoder, or as a drop-in ViT for your own multimodal training. ✨ Highlights - Codec-driven patchifier. Patches are selected by a per-patch importance map derived from the codec — motion vectors + P-frame residual energy for HEVC/H.265, or the learned rate map of the neural codec DCVC-RT. For a 64-frame clip it keeps all I-frame patches plus the top-k P-frame patches within a 4096-token budget (~75% token reduction). Chunk-wise and collage patchification are also supported. - Codec-agnostic. The same interface accepts a traditional codec (HEVC/H.265) or a neural codec (DCVC-RT) with no architecture or retraining change. - Trained from scratch. No billion-scale image-text ViT initialization — Mage-ViT is optimized with a large-scale cluster-discrimination objective on ~100M unlabeled images/videos. 🏗️ Architecture A 24-layer pre-norm Vision Transformer trunk (hidden size 1024, 16 attention heads, GELU MLP at 4× expansion) processes the variable-length token sequence produced by the codec-driven patchifier, followed by a multi-head attention pooling head. The standalone encoder in this repo consumes pixelvalues (and optional patchpositions); codec-driven patch selection is applied upstream in the Mage-VL data pipeline. Pre-training is a two-stage, from-scratch recipe in bf16: (1) variable-resolution image pre-training (224–448), then (2) joint image + video pre-training (video at resolution 2

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How embedding models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys mage-v for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (mage-v below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/embeddings \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"mage-v","input":"text to embed"}'

Details

Languages, data & research

Licence

Open weights

Sources

Weights & code

Explore

More embedding models