jina-embeddings-omni-small

jina-embeddings-v5-omni-small: Multi-Task Omni Embedding Base (Small)

ArXiv | Blog

Average score vs. parameter count across image (MIEB-Lite), video (MMEB-V), and audio (MAEB) benchmarks — jina-v5-omni-nano and jina-v5-omni-small define the open-weight frontier (Table 1 in the ArXiv report).

Model Overview

jina-embeddings-v5-omni-small is a multimodal embedding model that accepts text, images, video, and audio and produces embeddings in a shared vector space aligned with the text-only jinaai/jina-embeddings-v5-text-small — so you can index with text and query with any modality, no reindexing. For a more compact alternative, see jinaai/jina-embeddings-v5-omni-nano.

This is the base repository — it holds all task adapters (retrieval, classification, clustering, text-matching). For a single task, pre-merged task-specific variants are also available:

• jinaai/jina-embeddings-v5-omni-small-retrieval — query–document semantic search and RAG (raw-transformers users prepend Query: / Document: to text; sentence-transformers users call encode_query() / encode_document()).
• jinaai/jina-embeddings-v5-omni-small-classification — assigning labels via embedding similarity — zero-shot and few-shot classification across modalities.
• jinaai/jina-embeddings-v5-omni-small-clustering — grouping semantically similar items — topic discovery, deduplication, exploratory analysis.
• jinaai/jina-embeddings-v5-omni-small-text-matching — symmetric pairwise similarity scoring — STS, paraphrase and near-duplicate detection.

Via Elastic Inference Service

The fastest way to use v5-omni in production. Elastic Inference Service (EIS) provides managed embedding inference with built-in scaling, so you can generate embeddings directly within your Elastic deployment.

# Retrieve the configuration of the preconfigured omni-small inference endpoint
GET /_inference/embedding/.jina-embeddings-v5-omni-small

# Generate an embedding for a single piece of text using the predefined endpoint
POST _inference/embedding/.jina-embeddings-v5-omni-small
{
  "input": [
    "This is a test"
  ]
}

# Fuse a text description and an image into a single embedding via a multimodal content block
POST _inference/embedding/.jina-embeddings-v5-omni-small
{
  "input": [
    {
      "content": [
        { "type": "text",  "value": "A small blue square" },
        { "type": "image", "format": "base64", "value": "" }
      ]
    }
  ]
}

# Create a custom endpoint that truncates omni-small embeddings to 32 dimensions
PUT _inference/embedding/jina-omni-small-32d
{
  "service": "elastic",
  "service_settings": {
    "model_id": "jina-embeddings-v5-omni-small",
    "dimensions": 32
  }
}

See the Elastic Inference Service documentation for setup details.

Install

# core
pip install transformers torch pillow numpy

# Optional — install only the extras for the modalities you actually use:
pip install librosa soundfile      # audio decoding
pip install torchcodec                # video decoding for proc(videos=path) (transformers default backend)
pip install av                        # video decoding for SentenceTransformer model.encode("clip.mp4")
pip install pdf2image pypdfium2    # PDF rendering
pip install cairosvg pillow    # SVG rendering
pip install "vllm==0.20.1"        # high-throughput serving (validated)
pip install sentence-transformers  # one-call multimodal API

For minimum versions see the Requirements section below (transformers >= 4.57, torch >= 2.5; vLLM path validated with vllm == 0.20.1).

Quickstart

from PIL import Image
import librosa, torch
from transformers import AutoModel, AutoProcessor, WhisperFeatureExtractor

repo = "jinaai/jina-embeddings-v5-omni-small"
model = AutoModel.from_pretrained(repo, trust_remote_code=True, default_task="retrieval").eval()
proc  = AutoProcessor.from_pretrained(repo, trust_remote_code=True)

# model.embed(**inputs) returns L2-normalized last-token embeddings.
t_vec = model.embed(**proc(text="Query: Which planet is known as the Red Planet?", return_tensors="pt").to(model.device))
i_vec = model.embed(**proc(images=Image.open("photo.jpg"), text="", return_tensors="pt").to(model.device))
v_vec = model.embed(**proc(videos="clip.mp4", text="", return_tensors="pt").to(model.device))
# Needs torchcodec or torchvision. On Windows or envs without either, use model.encode("clip.mp4") from the sentence-transformers section below (av-only, no codec libs required).

# Audio has no string placeholder — build the prompt from config. Size the
# audio run from the REAL Whisper frame mask: the feature extra