llama-nemotron-rerank-vl

Model Overview

Description:

The llama-nemotron-rerank-vl-1b-v2 was developed by NVIDIA for multimodal question-answering retrieval. It is optimized for providing a logit score that represents how relevant a document page is to a given query. The model can process documents in the form of image, text, or image and text combined. The expected images are screenshots of document pages or slides. Documents are ranked given a user query in text form. The model supports images containing text, tables, charts, and infographics. We report the model's performance by evaluating it on the popular ViDoRe V1, V2 and the new Vidore V3 (see Vidore LB for details) multimodal retrieval benchmarks, and on two internally curated visual retrieval datasets.

The reranking model serves as a key component of a multimodal retrieval system, such as a vision RAG pipeline, where it helps improve overall accuracy. A multimodal retrieval system often uses a multimodal embedding model (dense) to return relevant documents given the input. A reranking model can be used to rerank the potential candidates into a final order. The reranking model takes the query and document pairs as input, and its self-attention can perform deeper interaction between their tokens. It’s not scalable to apply a ranking model to all documents in the knowledge base for a given query; therefore, ranking models are often deployed to rerank top candidate documents retrieved by embedding models.

This model is ready for commercial use.

Use Case:

The llama-nemotron-rerank-vl-1b-v2 is most suitable for users who want to build a multimodal question-and-answer application over a large corpus, leveraging the latest dense retrieval technologies.

License/Terms of Use:

This model and its associated configuration files are licensed under the OpenMDW License Agreement, version 1.1 (OpenMDW-1.1). Additional Information: Llama 3.2 Community Model License Agreement. Built with Llama.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Release Date

12/18/2025 via https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2

Model Architecture:

Architecture Type: Transformer Network Architecture: : Eagle VLM architecture with SigLIP 2 400M vision encoder and llama-nemotron-rerank-1b-v2 model as language model.

The llama-nemotron-rerank-vl-1b-v2 is a cross-encoder model with approximately 1.7B parameters. It is a fine-tuned version of an NVIDIA Eagle-family model, which consists of the SigLIP 2 400M vision encoder and the Llama 3.2 1B language model. The final embedding output by the decoder is aggregated using a mean pooling strategy, and a binary classification head is fine-tuned for the ranking task. The CrossEntropy loss is used to maximize the likelihood of (visual) documents containing information to answer the question and minimize the likelihood for (negative) documents that do not contain information to answer the question.

The vision-language model reranker incorporates key innovations from NVIDIA, including Eagle 2 work which uses a tiling-based VLM architecture, and nemoretriever-parse. The Eagle 2 architecture, available on Hugging Face, significantly enhances multimodal understanding through its dynamic tiling and mixture of vision encoders design. It particularly improves performance on tasks that involve high-resolution images and complex visual content.

Input(s):

Input Type(s): Image, Text

Input Format(s):

• Image: Red, Green, Blue (RGB)
• Text: String

Input Parameters:

• Image: Two-Dimensional (2D)
• Text: One-Dimensional (1D)

Other Properties Related to Input:

The model was fine-tuned exclusively on image data, using max_input_tiles = 4 and the maximum context length of 2048 tokens. For evaluation, it was tested on image-only, image+text, and text-only inputs, with max_input_tiles = 6 and the maximum context length of 10240 tokens. Inputs exceeding the maximum length are truncated.

Output(s)

Output Type(s): Floats

Output Format(s): List of Floats

Output Parameters: 1D

Other Properties Related to Output: Each value corresponds to a raw logit. Users can choose to apply a Sigmoid activation function to the logits to convert them into probabilities during model usage.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Installation

The model requires transformers version >= 4.56.0, and optionally flash-attention.

pip install "transformers>=4.56.0" pip install "flash-attn>=2.6.3,<2.8" --no-build-isolation

Usage

Using Sentence Transformers

Install Sentence Transformers:

pip install sentence_transformers

from sentence_transformers import CrossEncoder
from transformers.image_utils import load_image

model = CrossEncoder("nvidia/llama-nemotron-rerank-vl-1b-v2", trust_remote_code=True)

query = "How is AI improving the intelligence and capabilities of robots?"
documents = [
    "AI enables robots to perceive, plan, and act autonomously.",
    "AI is transforming autonomous vehicles by enabling safer, smarter, and more reliable decision-making on the road.",
    "A biological foundation model designed to analyze and generate DNA, RNA, and protein sequences.",
]
images = [
    load_image("https://developer.download.nvidia.com/images/isaa