Ovis-Omni-Embedding

Ovis-Omni-Embedding-3B is a 3B-parameter universal embedding model for text, images, visual documents, video, audio, and interleaved multimodal inputs. It maps every supported input type into one coherent representation space, enabling any-to-any retrieval with a single encoder.

The model is initialized from Qwen2.5-Omni-3B. Rather than attaching separate modality-specific embedding towers, it retains the native text tokenizer, vision encoder, audio encoder, and shared Thinker backbone. The speech-generation Talker and language-modeling head are removed, and the final-layer hidden state at the last non-padding token is used directly as the retrieval embedding.

Ovis-Omni-Embedding-3B is designed for multimodal search, retrieval-augmented generation, recommendation, visual-document retrieval, video and audio search, and agentic retrieval over tools, interfaces, and memory.

Architecture

Figure 2 from the technical report. Ovis-Omni-Embedding-3B encodes text, visual, and audio inputs as an interleaved token sequence processed by the Qwen2.5-Omni Thinker with TMRoPE. The final-layer hidden state at the last non-padding token becomes the retrieval embedding, with no modality-specific projection heads.

An input is formatted with a retrieval instruction through the native processor and chat template. Text, visual, and acoustic tokens are then processed as one interleaved sequence by the shared causal Transformer. Qwen2.5-Omni's time-aligned multimodal rotary position embedding preserves temporal alignment between audio and video.

Training

Figure 4 from the technical report. The training corpus spans text, images, video, audio, visual documents, and interleaved inputs. Homogeneous-source sampling forms each micro-batch from one dataset and deduplicates pooled candidates to create informative in-batch negatives without positive collisions.

Training follows three stages:

1. Omni-modal contrastive pretraining. Globally mixed candidates and cross-device in-batch negatives establish broad alignment. The objective combines difficulty-aware focal contrastive learning with similarity-distribution distillation from complementary modality experts.
2. Full-parameter homogeneous finetuning. High-quality downstream data refine fine-grained discrimination. Samples within each micro-batch come from one dataset, and candidate deduplication prevents positive collisions and false in-batch negatives.
3. Annealing Embedding Distillation. Teacher-correct examples are retained, unresolved student examples are upsampled, and confidence-adaptive forward-KL supervision transfers complementary expert capabilities without adding inference-time towers.

Figure 5 from the technical report. Embedding Distillation transfers similarity distributions from complementary experts, while the inference-time low-rank module combines a shared PCA basis with lightweight residual adapters for compact embeddings.

Retrieval interface

Ovis-Omni-Embedding-3B is a bi-encoder, not a cross-encoder:

1. Pair the query with the task instruction and format it with the model's native processor and chat template.
2. Encode queries and candidates independently.
3. Extract the final-layer hidden state at the last non-padding token.
4. L2-normalize the query and candidate embeddings.
5. Rank candidates by cosine similarity, equivalently the dot product of the normalized vectors.

No answer generation or query-candidate cross-attention is used during retrieval. Classification labels, passages, images, documents, videos, audio clips, and multimodal items are all treated as candidates in the same embedding space.

Performance

MMEB-v3

MMEB-v3 is an omni-modal benchmark comprising 190 datasets across image, video, visual-document, text, audio, and agent retrieval. Ovis-Omni-Embedding-3B achieves 58.46 overall, outperforming the strongest compared baseline by 5.19 points, and ranks first on the aggregate score of every modality group.

Across the 31 aggregate and sub-task entries in the complete comparison, Ovis-Omni-Embedding-3B ranks first on 22 and second on 8. MultiConIR is the only entry on which it falls outside the top two.

Scores are percentages and higher is better. Red marks the best result in each row, underlining marks the second best, and Ovis scores are bold. The overall score is the unweighted average over all 190 MMEB-v3 datasets. MMEB-v3 primarily uses Hit@1 for image, video, audio, and agent tasks and nDCG@5 for text and visual-document retrieval.

Additional benchmark results

These benchmark families use their own official aggregation procedures, so their scores should not be averaged together. MAEB and MVEB results are local evaluations inserted into the corresponding leaderboard snapshots, as described in the technical report.

Embedding dimensions

The native output width is 2048 because no embedding projection head is added to the backbone. For deployments with tighter storage or latency budgets, the post-hoc elastic-dimension module supports 1024, 512, 256, and 128 dimensions. It combines a shared, modality-balanced PCA rotation with a zero-initialized residual linear adapter and fo