[📃 Paper] [📂 GitHub]

MoE-ViE is a family of Mixture-of-Experts vision encoders for image and video understanding, trained with a contrastive vision-language recipe. Each MoE block keeps a shared always-on expert alongside a routed pool, so only a small fraction of the weights is used for any given token.

Model Developer: Meta

This checkpoint: MoEViE-B16-224 — B/16 at 224px.

Model Configurations

MoE-ViE comes in three sizes. Only active experts run per token, so the activated parameter count is far below the total.

Every MoE block has 1 shared expert that is always active plus a routed pool; the active/total column counts the shared expert. Routing is per-token top-k on a sigmoid gate.

Model Performance

Zero-shot results. Top-1 accuracy (%) for classification, recall@1 (%) for retrieval.

How to use

Install

git clone https://github.com/facebookresearch/moe_vie
cd moe_vie
pip install -r requirements.txt

Requires a CUDA GPU — the Mixture-of-Experts kernels are compiled with Triton at runtime.

Image and text feature extraction

import torch
from PIL import Image
from open_clip import create_model_and_transforms, get_tokenizer, image_to_device

MEAN, STD = (0.5, 0.5, 0.5), (0.5, 0.5, 0.5)

model, _, preprocess = create_model_and_transforms(
    "MoEViE-B16-224",
    pretrained=True,   # downloads from the Hub
    force_preprocess_cfg=dict(
        patch_size=16, size_range=(224, 224), center_crop=True, window_size=1
    ),
    image_mean=MEAN, image_std=STD,
)
model = model.cuda().eval()
tokenizer = get_tokenizer("MoEViE-B16-224")

labels = ["a diagram", "a dog", "a cat"]
packed, _ = preprocess.collate_fn([(preprocess(Image.open("cat.png").convert("RGB")), 0)])
packed = image_to_device(packed, "cuda", torch.float32, mean=MEAN, std=STD)
text = tokenizer(labels).cuda()

with torch.no_grad(), torch.autocast("cuda"):
    image_features = model.encode_image(packed, normalize=True)
    text_features = model.encode_text(text, normalize=True)
    probs = (model.logit_scale.exp() * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", probs)

See demo/demo.py and the repository README for the zero-shot evaluation suite.

License

Released under CC BY-NC 4.0 — non-commercial research use.

Citation

If you find this work useful, please cite:

@article{zhang2026moevie,
  title={MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding},
  author={Bonan Zhang and Shiyu Dong and Quan Hung Tran and Katharina Gschwind and Shuqi Yang and Sijia Chen and Adel Ahmadyan and Seungwhan Moon and Lu Zhang and Ahmed Kirmani and Babak Damavandi and Anuj Kumar},
  journal={arXiv preprint arXiv:2608.17402},
  year={2026}
}