UniVR-Planning

Model Summary

UniVR is the first framework that simultaneously learns complex reasoning, fine-grained physical dynamics, and long-term planning from pure visual demonstrations — without relying on dense image-text pairs or task-specific heuristics.

Built on Emu3.5 (34B), UniVR uses a unified next-token prediction objective to directly generate visual reasoning traces given an image and instruction. Training employs a two-stage pipeline: supervised cold initialization on the VR-X dataset, followed by VR-GRPO reinforcement learning with complementary global and step-focal rewards.

Available Checkpoints

Both checkpoints live as subdirectories of this repo: ByteDance/UniVR-34B-Planning.

Key Results

VR-X Benchmark

UniVR achieves up to 25% improvement over the Emu3.5 baseline and approaches Gemini 3 Pro + Nano Banana 2 with only 34B parameters.

Multimodal Understanding

Enhanced visual reasoning also boosts standard multimodal benchmarks — no degradation of the base model's capabilities.

Quick Start

Installation

git clone https://github.com/bytedance/UniVR.git
cd UniVR
bash install.sh

Inference

cd UniVR_SFT

# Download checkpoint (choose Planning or General subdir)
huggingface-cli download ByteDance/UniVR-34B-Planning --include "Planning/*" --local-dir weights/UniVR-34B-Planning

# Download VisionTokenizer
huggingface-cli download BAAI/Emu3.5-VisionTokenizer --local-dir weights/Emu3.5-VisionTokenizer

# Run inference
bash scripts/inference.sh

Configure configs/config.py to set model paths and prompts:

{
    "prompt": "Tie the red rope around the white gift box. Finish this task in 3 steps.",
    "reference_image": "path/to/first_frame.jpg",
}

Training

SFT (Cold Initialization):

cd UniVR_SFT
# LoRA (2 nodes × 8 GPUs)
bash scripts/train_sft_lora.sh
# Full parameter (4 nodes × 8 GPUs)
bash scripts/train_sft_full.sh

RL (VR-GRPO):

cd UniVR_RL
bash examples/emu3_grpo_lora.sh

Method: VR-GRPO

UniVR proposes VR-GRPO (Visual Reasoning GRPO), a reinforcement learning paradigm that combines:

• Global Reward (R_g): A VLM evaluator assesses overall task completion and visual quality via pairwise comparison.
• Step-Focal Reward (R_s): Identifies the most error-prone sub-steps by computing inter-trajectory CLIP-feature variance across rollout samples, then applies fine-grained VLM evaluation on critical windows.
• Combined Reward: R_reason = R_g − λ|R_g − R_s|, enforcing both terminal correctness and procedural integrity.

This design prevents reward hacking in long-horizon tasks where global-only rewards overlook intermediate physical violations and logical gaps.

Sample Outputs

Training Data

UniVR is trained on VR-X, a large-scale benchmark curated from 1.5M raw samples across 16 diverse sources:

Download: ByteDance/VR-X-SFT-RL

Citation

@article{ren2026univr,
  title={UniVR: Thinking in Visual Space for Unified Visual Reasoning},
  author={Zhongwei Ren and Yunchao Wei and Zhao Yao and Guixun Luo and Yao Zhao and Weibo Gong and Xiao Liu and Anran Wang and Xiangtai Li and Xiaojie Jin},
  url={https://arxiv.org/abs/2607.12800},
  year={2026},
}

License

This project is released under the CC BY 4.0 License.

Acknowledgements

UniVR is built upon Emu3.5 and verl. We thank the authors for their excellent open-source contributions.