LCO-Embedding-Omni-2605

Main Benchmarks (text, image, audio):

Other Capabilities (code, video)

Checkpoint Overview

Usage

Using Sentence Transformers

Install Sentence Transformers with the multimodal extras (for image, audio, and video support):

pip install "sentence_transformers[image,audio,video]" "transformers>=5.6.0"

import torch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "LCO-Embedding/LCO-Embedding-Omni-3B-2605",
    model_kwargs={
        "dtype": torch.bfloat16,
        # "attn_implementation": "flash_attention_2",  # recommended, if a flash-attn build exists for your platform
    },
)

The same Summarize the above in one word: instruction used in the paper is baked into the chat template, so encode() takes plain text, file paths, URLs, or multimodal dicts directly.

Text Retrieval

query = "What is the tallest mountain in the world?"
documents = [
    "Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. Its elevation of 8,848.86 metres was established by a joint Chinese-Nepali survey in 2020.",
    "K2, at 8,611 metres above sea level, is the second-highest mountain on Earth, after Mount Everest. It lies in the Karakoram range on the China-Pakistan border.",
    "Mount Kilimanjaro is a dormant volcano in Tanzania. It is the highest mountain in Africa, with its summit about 5,895 metres above sea level.",
]

query_embedding = model.encode(query)
document_embeddings = model.encode(documents)
print(model.similarity(query_embedding, document_embeddings))
# tensor([[0.5368, 0.5053, 0.4989]])

Image Retrieval

query = "How many input modalities does Qwen2.5-Omni support?"
documents = [
    "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/qwen2.5omni_hgf.png",
    "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/llama4_hgf.png",
]

query_embedding = model.encode(query)
document_embeddings = model.encode(documents, batch_size=1)
print(model.similarity(query_embedding, document_embeddings))
# tensor([[0.6544, 0.3852]])

Audio Retrieval

query = "A light piano piece"
documents = [
    "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/joe_hisaishi_summer.mp3",
    "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/jay_chou_superman_cant_fly.mp3",
]

query_embedding = model.encode(query)
document_embeddings = model.encode(documents, batch_size=1)
print(model.similarity(query_embedding, document_embeddings))
# tensor([[0.3649, 0.0662]])

Video Retrieval

# For video on smaller GPUs, cap the processor up front:
model[0].processing_kwargs.update({
    "video": {"max_pixels": 64 * 28 * 28, "do_sample_frames": True, "fps": 1},
})

query = "How to cook Mapo Tofu?"
documents = [
    "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/mapo_tofu.mp4",
    "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/zhajiang_noodle.mp4",
]

query_embedding = model.encode(query)
document_embeddings = model.encode(documents, batch_size=1)
print(model.similarity(query_embedding, document_embeddings))
# tensor([[0.6408, 0.4967]])

Multimodal Inputs

To embed a document that combines multiple modalities, pass a dict with any combination of "text", "image", "audio", and "video" keys instead of a single path or string:

documents = [
    {
        "text": "A cooking tutorial for Mapo Tofu",
        "video": "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/mapo_tofu.mp4",
    },
    {
        "image": "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/qwen2.5omni_hgf.png",
        "audio": "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/joe_hisaishi_summer.mp3",
    },
]
document_embeddings = model.encode(documents, batch_size=1)
print(document_embeddings.shape)
# (2, 2048)

The expected outputs above were produced in bfloat16 on a CUDA device with the default (sdpa) attention. Exact values shift slightly in the fourth decimal with a different dtype or attention implementation.

Using Transformers

All inference code is the same with our OG models and can seamlessly support the new checkpoint by changing the model name.

Contributors

LCO-Embedding Team members that made this release happen:

Chenghao Xiao, Ruifeng Yuan, Long Li, Fengyu Cai, Yiqi Liu, Yang Wang, Chenghua Lin, Hao Zhang, Hou Pong Chan, Ling Zhang