gme-Qwen2-VL

GME-Qwen2-VL-2B

We are excited to present GME-Qwen2VL series of unified multimodal embedding models, which are based on the advanced Qwen2-VL multimodal large language models (MLLMs).

The GME models support three types of input: text, image, and image-text pair, all of which can produce universal vector representations and have powerful retrieval performance.

Key Enhancements of GME Models:

• Unified Multimodal Representation: GME models can process both single-modal and combined-modal inputs, resulting in a unified vector representation. This enables versatile retrieval scenarios (Any2Any Search), supporting tasks such as text retrieval, image retrieval from text, and image-to-image searches.
• High Performance: Achieves state-of-the-art (SOTA) results in our universal multimodal retrieval benchmark (UMRB) and demonstrate strong evaluation scores in the Multimodal Textual Evaluation Benchmark (MTEB).
• Dynamic Image Resolution: Benefiting from Qwen2-VL and our training data, GME models support dynamic resolution image input.
• Strong Visual Retrieval Performance: Enhanced by the Qwen2-VL model series, our models excel in visual document retrieval tasks that require a nuanced understanding of document screenshots. This capability is particularly beneficial for complex document understanding scenarios, such as multimodal retrieval-augmented generation (RAG) applications focused on academic papers.

Developed by: Tongyi Lab, Alibaba Group

Paper: GME: Improving Universal Multimodal Retrieval by Multimodal LLMs

Model List

Usage

Transformers

The remote code has some issues with transformers>=4.52.0, please downgrade or use sentence_transformers

from transformers import AutoModel
from transformers.utils.versions import require_version

require_version(
    "transformers<4.52.0",
    "The remote code has some issues with transformers>=4.52.0, please downgrade: pip install transformers==4.51.3"
)

t2i_prompt = 'Find an image that matches the given text.'
texts = [
    "The Tesla Cybertruck is a battery electric pickup truck built by Tesla, Inc. since 2023.",
    "Alibaba office.",
]
images = [
    'https://upload.wikimedia.org/wikipedia/commons/e/e9/Tesla_Cybertruck_damaged_window.jpg',
    'https://upload.wikimedia.org/wikipedia/commons/e/e0/TaobaoCity_Alibaba_Xixi_Park.jpg',
]

gme = AutoModel.from_pretrained(
    "Alibaba-NLP/gme-Qwen2-VL-2B-Instruct",
    torch_dtype="float16", device_map='cuda', trust_remote_code=True
)

# Single-modal embedding
e_text = gme.get_text_embeddings(texts=texts)
e_image = gme.get_image_embeddings(images=images)
print('Single-modal', (e_text @ e_image.T).tolist())
## Single-modal [[0.359619140625, 0.0655517578125], [0.04180908203125, 0.374755859375]]

# How to set embedding instruction
e_query = gme.get_text_embeddings(texts=texts, instruction=t2i_prompt)
# If is_query=False, we always use the default instruction.
e_corpus = gme.get_image_embeddings(images=images, is_query=False)
print('Single-modal with instruction', (e_query @ e_corpus.T).tolist())
## Single-modal with instruction [[0.429931640625, 0.11505126953125], [0.049835205078125, 0.409423828125]]

# Fused-modal embedding
e_fused = gme.get_fused_embeddings(texts=texts, images=images)
print('Fused-modal', (e_fused @ e_fused.T).tolist())
## Fused-modal [[1.0, 0.05511474609375], [0.05511474609375, 1.0]]

sentence_transformers

The encode function accept str or dict with key(s) in {'text', 'image', 'prompt'}.

Do not pass prompt as the argument to encode, pass as the input as a dict with a prompt key.

from sentence_transformers import SentenceTransformer

t2i_prompt = 'Find an image that matches the given text.'
texts = [
    "The Tesla Cybertruck is a battery electric pickup truck built by Tesla, Inc. since 2023.",
    "Alibaba office.",
]
images = [
    'https://upload.wikimedia.org/wikipedia/commons/e/e9/Tesla_Cybertruck_damaged_window.jpg',
    'https://upload.wikimedia.org/wikipedia/commons/e/e0/TaobaoCity_Alibaba_Xixi_Park.jpg',
]

gme_st = SentenceTransformer("Alibaba-NLP/gme-Qwen2-VL-2B-Instruct")

# Single-modal embedding
e_text = gme_st.encode(texts, convert_to_tensor=True)
e_image = gme_st.encode([dict(image=i) for i in images], convert_to_tensor=True)
print('Single-modal', (e_text @ e_image.T).tolist())
## Single-modal [[0.356201171875, 0.06536865234375], [0.041717529296875, 0.37890625]]

# How to set embedding instruction
e_query = gme_st.encode([dict(text=t, prompt=t2i_prompt) for t in texts], convert_to_tensor=True)
# If no prompt, we always use the default instruction.
e_corpus = gme_st.encode([dict(image=i) for i in images], convert_to_tensor=True)
print('Single-modal with instruction', (e_query @ e_corpus.T).tolist())
## Single-modal with instruction [[0.425537109375, 0.1158447265625], [0.049835205078125, 0.413818359375]]

# Fused-modal embedding
e_fused = gme_st.encode([dict(text=t, image=i) for t, i in zip(texts, images)], convert_to_tensor=True)
print('Fused-modal', (e_fused @ e_fused.T).tolist())
## Fused-modal [[0.99951171875, 0.0556640625], [0.0556640625, 0.99951171875]]

Evaluation

We validated the performance on our universal multimodal retrieval benchmark (UMRB, see Release UMRB) among others.

| | | Single-modal |