jina-embeddings-omni-small-retrieval

jina-embeddings-v5-omni-small-retrieval: Retrieval-Targeted Omni Embedding (Small)

ArXiv | Blog

Average score vs. parameter count across image (MIEB-Lite), video (MMEB-V), and audio (MAEB) benchmarks — jina-v5-omni-nano and jina-v5-omni-small define the open-weight frontier (Table 1 in the ArXiv report).

Model Overview

jina-embeddings-v5-omni-small-retrieval is a multimodal embedding model that accepts text, images, video, and audio and produces embeddings in a shared vector space aligned with the text-only jinaai/jina-embeddings-v5-text-small-retrieval — so you can index with text and query with any modality, no reindexing. For a more compact alternative, see jinaai/jina-embeddings-v5-omni-nano-retrieval.

This is the retrieval-targeted variant of the jina-embeddings-v5-omni-small family. The combined repo containing all task adapters is jinaai/jina-embeddings-v5-omni-small.

Via Elastic Inference Service

The fastest way to use v5-omni in production. Elastic Inference Service (EIS) provides managed embedding inference with built-in scaling, so you can generate embeddings directly within your Elastic deployment.

# Retrieve the configuration of the preconfigured omni-small inference endpoint
GET /_inference/embedding/.jina-embeddings-v5-omni-small

# Generate an embedding for a single piece of text using the predefined endpoint
POST _inference/embedding/.jina-embeddings-v5-omni-small
{
  "input": [
    "This is a test"
  ]
}

# Fuse a text description and an image into a single embedding via a multimodal content block
POST _inference/embedding/.jina-embeddings-v5-omni-small
{
  "input": [
    {
      "content": [
        { "type": "text",  "value": "A small blue square" },
        { "type": "image", "format": "base64", "value": "" }
      ]
    }
  ]
}

# Create a custom endpoint that truncates omni-small embeddings to 32 dimensions
PUT _inference/embedding/jina-omni-small-32d
{
  "service": "elastic",
  "service_settings": {
    "model_id": "jina-embeddings-v5-omni-small",
    "dimensions": 32
  }
}

See the Elastic Inference Service documentation for setup details.

Install

# core
pip install transformers torch pillow numpy

# Optional — install only the extras for the modalities you actually use:
pip install librosa soundfile      # audio decoding
pip install torchcodec                # video decoding for proc(videos=path) (transformers default backend)
pip install av                        # video decoding for SentenceTransformer model.encode("clip.mp4")
pip install pdf2image pypdfium2    # PDF rendering
pip install cairosvg pillow    # SVG rendering
pip install "vllm==0.20.1"        # high-throughput serving (validated)
pip install sentence-transformers  # one-call multimodal API

For minimum versions see the Requirements section below (transformers >= 4.57, torch >= 2.5; vLLM path validated with vllm == 0.20.1).

Quickstart

from PIL import Image
import librosa, torch
from transformers import AutoModel, AutoProcessor, WhisperFeatureExtractor

repo = "jinaai/jina-embeddings-v5-omni-small-retrieval"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
proc  = AutoProcessor.from_pretrained(repo, trust_remote_code=True)

# model.embed(**inputs) returns L2-normalized last-token embeddings.
q_vec = model.embed(**proc(text="Query: Which planet is known as the Red Planet?", return_tensors="pt").to(model.device))
d_vec = model.embed(**proc(text="Document: Mars is often referred to as the Red Planet due to its reddish appearance.", return_tensors="pt").to(model.device))

# Equivalent high-level call — handles the Query:/Document: prefix for you:
q_vec = model.encode(["Which planet is known as the Red Planet?"], task="retrieval", prompt_name="query")
d_vec = model.encode(["Mars is often referred to as the Red Planet due to its reddish appearance."], task="retrieval", prompt_name="document")
i_vec = model.embed(**proc(images=Image.open("photo.jpg"), text="", return_tensors="pt").to(model.device))
v_vec = model.embed(**proc(videos="clip.mp4", text="", return_tensors="pt").to(model.device))
# Needs torchcodec or torchvision. On Windows or envs without either, use model.encode("clip.mp4") from the sentence-transformers section below (av-only, no codec libs required).

# Audio has no string placeholder — build the prompt from config. Size the
# audio run from the REAL Whisper frame mask: the feature extractor pads every
# clip out to the 30s window, so counting padded frames emits 750 audio tokens
# for a 2s clip and a 30s clip alike, which is not how the model was trained.
# (The sentence-transformers path below does all of this for you.)
audio, _ = librosa.load("speech.wav", sr=16000)
feats = WhisperFeatureExtractor(feature_size=128)(
    audio, sampling_rate=16000, return_tensors="pt",
    padding="max_length", return_attention_mask=True,