Qwen-Image-Bench

A fine-tuned judge model for evaluating text-to-image (T2I) generation quality. Built on top of Qwen3.6-27B, it scores generated images across 5 hierarchical dimensions using structured checklists and outputs JSON-formatted evaluation results.

Links

Model Description

Q-Judger is a vision-language model fine-tuned specifically for automated evaluation of text-to-image generated images. Given a text prompt and a generated image, the model evaluates the image on fine-grained quality criteria organized in a 3-level hierarchy and outputs structured JSON scores.

• Base Model: Qwen3.6-27B
• Task: Image quality evaluation / judging
• Input: Text prompt + generated image
• Output: Structured JSON with per-dimension scores (0 = Fail, 1 = Pass, 2 = Excel, N/A)
• Thinking Mode: Enabled — the model uses chain-of-thought reasoning before producing the final JSON output

Evaluation Dimensions

The model evaluates images across 5 top-level dimensions, each with multiple sub-dimensions:

Quality

• Realism: Physical Logic, Material Texture
• Detail: Noise, Edge Clarity, Naturalness
• Resolution: Resolution

Aesthetics

• Composition: Composition
• Color Harmony: Color Harmony
• Lighting: Lighting & Atmosphere
• Anatomical Portraiture: Anatomical Fidelity
• Emotional Expression: Emotional Expression
• Style Control: Style Control

Alignment

• Attributes: Quantity, Facial Expression, Material Properties, Color, Shape, Size
• Actions: Contact Interaction, Non-contact Interaction, Full-body Action
• Layout: 2D Space, 3D Space
• Relations: Composition Relationship, Difference/Similarity, Containment
• Scene: Real-world Scene, Virtual Scene

Real-world Fidelity

• Fairness: Social Bias, Cultural Fairness
• Safety & Compliance: Safety & Compliance
• World Knowledge: Animals, Objects, Information Visualization, Temporal Characteristics, Cultural Elements

Creative Generation

• Imagination: Imagination
• Feature Matching: Feature Matching
• Logical Resolution: Logical Resolution
• Text Rendering: Text Accuracy, Text Layout, Font, Cross-lingual Generation
• Design Applications: Graphic Design, Product Design, Spatial Design, Fashion Styling, Game Design, Art Design
• Visual Storytelling: Cinematic Style, Camera / Lens Style, Storyboard Creation, Shot Sizes, Composition, Angles, Comic Creation

Scoring Methodology

Raw Score Mapping

Aggregation

1. Level-3 → Level-2: Average all non-N/A Level-3 scores within a Level-2 category
2. Level-2 → Level-1: Average all Level-2 scores within a Level-1 dimension
3. Level-1 → Total: Average all Level-1 dimension scores

Human Agreement

We validate the judge model against human expert rankings by computing Spearman rank correlation ($\rho$) between the model's rankings and human expert rankings across the five L1 pillars and overall. All correlations are statistically significant ($p < 10^{-4}$, $N = 18$ models).

Quick Start

Get the Inference Code

git clone https://github.com/QwenLM/Qwen-Image-Bench.git
cd Qwen-Image-Bench

Installation

1. Create and activate a virtual environment with uv:

uv venv myenv --python 3.11
source myenv/bin/activate

2. Install PyTorch (select the command matching your CUDA version):

See the official guide: https://pytorch.org/get-started/locally/

3. Install Python dependencies:

uv pip install -r requirements.txt

This installs all required dependencies including ms-swift.

Run Inference

python judge.py \
  --input your_data.jsonl \
  --model Qwen/Qwen-Image-Bench

Input Format

Prepare a CSV, JSON, or JSONL file with the following columns:

Output Format

The model outputs a JSON object per dimension, structured as:

{
  "Level-2 Dimension": {
    "Level-3 Dimension": {"score": 0|1|2|"N/A"}
  }
}

Example (Quality dimension):

{
  "Realism": {
    "Physical Logic": {"score": 1},
    "Material Texture": {"score": 2}
  },
  "Detail": {
    "Noise": {"score": 1},
    "Edge Clarity": {"score": 1},
    "Naturalness": {"score": 1}
  },
  "Resolution": {
    "Resolution": {"score": 2}
  }
}

CLI Options