CASA-Helium1-VL

CASA ([Project Page][blog] . [arXiv][casa-arxiv] . [github][casa-git]) stands for Cross-Attention over Self-Attention. CASA is a VLM based entirely on cross-attention (CA) with enjoys practical benefits for streaming tasks by never adding image tokens to the KVCache of the text model.

Specifically, CASA layers inject visual tokens into a text stream by using image-to-text cross-attention, which act in smaller local attention windows. Simply put, at inference, the model only looks at the latest seen image explicitly. We show that this setup can be naturally leveraged streaming inference e.g. on video inputs, enabling almost constant memory time and inference cost For qualitative samples of CASA used for live video captioning, please check the associated HuggingFace space.

Model Details

Model Description

This model page contains the model weights for CASA trained from a pretrained text-only Helium1-2B backbone and from the image encoder from Qwen2.5-VL-3B. In the collection, we also provides weights for:

• CASA-Qwen2_5-VL-3B: A CASA model adapted from the full pretrained Qwen2.5-VL-3B (keeping the backbone LLM weights are kept frozen)
• CASA-Qwen2_5-VL-3B-LiveCC: A CASA model adapted from the full pretrained Qwen2.5-VL-3B and futher finetuned for live video captioning.
• Helium1-VL-2B: A reference VLM trained from Helium1-2B with standard token insertion mechanism in the same setting as CASA-Helium1-VL-2B.

Model Summary:

• Developed by: Kyutai
• Model type: Multimodal vision+text model based on Cross-Attention
• Language(s) (NLP): English
• License: CC-BY-NC-SA-4.0
• LLM Backboner from: Helium1 2B
• Image Encoder from: Qwen2.5-VL 3B
• Terms of use: As the released models include frozen weights of the Qwen2.5VL-3B image encoder, the weights are subject to the Qwen RESEARCH LICENSE AGREEMENT

Model Sources

• Project Page [kyutai.org/casa][blog]
• Preprint [arXiv][casa-arxiv]
• Repository: [Github kyutai-labs/casa][casa-git]

Uses

Direct Use

The intended use of the Helium model is research and development of vision-language systems, including but not limited to image or video understanding.

CASA-Helium1-VL-2B, Helium1-VL-2B and CASA-Qwen2_5-VL-2B can be used as vision-language models to analyze or interpret images as input signals.

CASA-Qwen2_5-VL-2B-LiveCC can be used as a vision-language model on streaming videos as inputs at 2fps.

The models can be used primarly with English as a language. For most downstream use cases, the model should be aligned with supervised fine-tuning, RLHF or related methods.

Out-of-Scope Use

The model should not be used in other languages than the ones on which it was trained. The model is not intended to be used to impersonate other people or any malicious use of any kind.

Bias, Risks, and Limitations

Our CASA-Helium1 model was not aligned to human preferences. As such, the model can generate incorrect, biased, harmful or generally unhelpful content. Thus, the model should not be used for downstream applications without further alignment, evaluations and mitigations of risks.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

See our [github repository][casa-git] for additional scripts to perform benchmark evaluation and live video captioning.

Below is a short snippet to show you how to load our models, process inputs, and run inference, using a standard HuggingFace transformers pipeline and chat template.

# Minimal requirements:
# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "rich",
#     "einops>=0.8.1",
#     "torch==2.7.0",
#     "transformers==4.51.3",
#     "torchvision==0.22.0",
#     "flash-attn @ https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.0.post2/flash_attn-2.8.0.post2+cu12torch2.7cxx11abiTRUE-cp310-cp310-linux_x86_64.whl"
# ]
# ///
import torch
from transformers.models.auto.modeling_auto import AutoModel
from transformers.models.auto.processing_auto import AutoProcessor

model_id = "kyutai/CASA-Helium1-VL-2B"
model = AutoModel.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    trust_remote_code=True,
).cuda()
processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "assets/casa_model.png",
            },
            {
                "type": "text",
                "text": "Describe this image.",
            },
        ],
    },
]
inputs = processor.tokenize_messages(messages=conversation)
inputs = inputs.to(model.device)
input_len = inputs["input_ids"].shape[1]
output_ids = model.generate_from_image(
  **inputs,
  max_new_tokens=512,
  pre_image_tokens=processor.pre_image_tokens,
  post_image_tokens=processor.post_image_tokens,
  eos_token_id=model.generation_config.eos_token_id,
)[0, input_len:]
response = processor.tokenizer.decode(output_ids, skip_special_tokens=True)
print(response)

Training Details

Please have a look at our associated [research paper][casa-arxiv] for details on the training pipeline.

Training Data

To train our CASA-Helium models we use the FineVision dataset as well as a small, non overlapping, sub