MolmoMotion-H3-F30

MolmoMotion is a 4B vision-language model that forecasts 3D point trajectories under natural-language action instructions. Given a short RGB observation history, a set of user-specified 2D query points with their 3D history, and an action description, it predicts where those points move in 3D (camera frame, in meters) over a future horizon.

This checkpoint is the autoregressive (AR) variant with history H=3 and future horizon F=30. Pick this variant for typical video use — three history frames, predicting roughly two seconds of motion at 15 fps.

Quick Links

• 📂 Models: MolmoMotion collection
• 📂 Dataset: allenai/molmo-motion-1m
• 📊 Benchmark: allenai/PointMotionBench
• 💻 Code: allenai/molmo-motion
• 🎥 Blog Post: MolmoMotion
• 📄 Paper: coming soon

Intended Use

MolmoMotion forecasts 3D point trajectories from RGB video, 2D query points, and a language action. Use it for trajectory-prediction / motion-forecasting research and as a starting point for downstream finetuning (e.g. the robot pick-and-place recipe in robotics/).

Pick the variant that matches your input:

Usage

Install the package, then load this checkpoint:

pip install git+https://github.com/allenai/molmo-motion.git

import torch
from molmo_motion import MolmoMotion, MolmoMotionProcessor

CKPT = "allenai/MolmoMotion-4B-H3-F30"
processor = MolmoMotionProcessor.from_pretrained(CKPT)
model = MolmoMotion.from_pretrained(CKPT)
model._internal = model._internal.to(torch.bfloat16).cuda()  # 4B params

See examples/01_quickstart.py for a full forward pass (building inputs, predict_trajectory, and visualizing the predicted trajectory).

Responsible Use

MolmoMotion is a research model that predicts point trajectories; it is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines. Predicted trajectories are model estimates and should be validated before being used to drive any downstream actuated system.

Disclaimer

MolmoMotion is trained on MolmoMotion-1M, which includes captions and trajectory annotations based on coordinates of objects in videos. The videos are provided by the owners of the datasets listed in allenai/molmo-motion-1m and allenai/PointMotionBench, and are subject to their respective license terms and use restrictions. Users who access videos from these sources to reconstruct any portion of the training data are responsible for reviewing and confirming that their intended use complies with the terms and conditions.

Acknowledgement

We thank Ropedia for providing videos from the Xperience dataset that were used in our research and development of MolmoMotion. The Xperience dataset is governed by Ropedia’s terms and conditions.

Citation

@misc{zhang2026molmomotionforecastingpointtrajectories,
    title={MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction},
    author={Jianing Zhang and Chenhao Zheng and Yajun Yang and Max Argus and Rustin Soraki and Winson Han and Taira Anderson and Chun-Liang Li and Shuo Liu and Jiafei Duan and Zhongzheng Ren and Jieyu Zhang and Ranjay Krishna},
    year={2026},
    eprint={2606.18558},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2606.18558},
}