Cosmos-H-Dreams

Description

Cosmos-H-Dreams is a real-time, action-conditioned generative surgical world model that lets a human operator or a learned surgical-robotics policy act inside a synthesized surgical scene and observe the interactions live. Given a surgical context frame and a stream of robot kinematic actions, the model autoregressively generates the resulting future video in short blocks, streaming interactively on a single GPU.

Unlike the bidirectional, offline Cosmos-H-Surgical-Simulator, Cosmos-H-Dreams is a causal, few-step self-forcing distilled student: it is distilled from a bidirectional teacher into a streaming model that responds immediately to actions, turning a passive video generator into a controllable surgical simulator. The released checkpoint specializes the model to da Vinci Research Kit (dVRK) tabletop suturing.

The model is intended for real-time surgical-skills rehearsal, interactive demonstration, closed-loop evaluation of surgical robotics policies, and synthetic data generation.

The released model is derived from the public NVIDIA Cosmos-Predict2.5-2B world foundation model for physical AI, and its teacher is warm-started from Cosmos-H-Surgical-Simulator (the Open-H 44D action-conditioned checkpoint).

This model is ready for commercial or non-commercial use.

License/Terms of Use

Use of this model is governed by the NVIDIA Open Model License Agreement.

Deployment Geography

Global

Use Case

Primarily intended for surgical robotics researchers, healthcare AI developers, academic institutions, and surgical robotics companies exploring interactive surgical simulation, surgeon/trainee rehearsal, closed-loop surgical policy evaluation, and synthetic data generation. Because it is interactive, the same served model can be driven live by a human (browser keyboard or Meta Quest headset) or by a learned policy in a closed-loop evaluation harness.

Release Date

• GitHub: 07/23/2026 via https://github.com/isaac-for-healthcare/Cosmos-H-Dreams
• Huggingface: 07/23/2026 via https://huggingface.co/nvidia/Cosmos-H-Dreams

Reference(s)

Ali, A., Bai, J., Bala, M., Balaji, Y., Blakeman, A., Cai, T., Cao, J., Cao, T., Cha, E., Chao, Y.-W., Chattopadhyay, P., Chen, M., Chen, Y., Cheng, S., Cui, Y., Diamond, J., Ding, Y., Fan, J., Fan, L., Feng, L., Ferroni, F., Fidler, S., Fu, X., Gao, R., Ge, Y., Gu, J., … Zhu, Y. (2025). World Simulation with Video Foundation Models for Physical AI (arXiv:2511.00062) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2511.00062

Link to Cosmos’ nvidia/Cosmos-Predict2.5-2B-Video2World Model Card

Link to nvidia/Cosmos-H-Surgical-Simulator Model Card (the teacher warm-start checkpoint).

Model Architecture

Architecture Type: Diffusion Transformer Network Architecture: Latent video diffusion transformer (DiT-style denoiser) with cross-attention conditioning, distilled into a causal, few-step autoregressive student with a streaming key/value (KV) cache.

This model was developed based on Cosmos-Predict2.5-2B-Video2World.

Cosmos-H-Dreams extends Cosmos-Predict2.5-2B-Video2World, a 2B-parameter diffusion transformer for video generation in latent space (Wan2.1 video tokenizer, spatial compression 8×, temporal compression 4×, 16 latent channels; hidden width 2048, 28 blocks, 16 heads, 2×2 spatial patchification, rotary position embeddings, AdaLN-LoRA modulation, Cosmos-Reason text encoder cross-attention). It incorporates two small MLPs that condition the model on kinematic actions through the timestep/AdaLN modulation pathway.

The model consumes a unified 44-dimensional action vector; each latent frame folds the 4 action steps it is responsible for into a 4 × 44 = 176-dimensional input. The unified action space lets a subset of the 44 dimensions carry an embodiment's native content while the remainder are zero-padded, which keeps the weights embodiment- and horizon-invariant. The released dVRK tabletop checkpoint uses a 20-dimensional dual-arm content vector (per patient-side manipulator: a 3D translation delta, a 6D continuous rotation, and a gripper value) zero-padded to 44D.

The model is built in two stages from the same backbone:

• A bidirectional teacher that learns action-conditioned surgical dynamics (warm-started from the Cosmos-H-Surgical-Simulator Open-H 44D checkpoint and fine-tuned on the target data, optionally at progressively longer temporal horizons, i.e., 13 → 25 → 49 → 73 frames, for long-rollout stability).
• A distilled causal streaming student (the released model) that subclasses the same backbone and adds causal temporal attention masking, a streaming KV cache with a rolling local window and an appearance "sink" frame, and CUDA-graph capture of the steady-state forward pass. The student rolls forward in outer blocks of 12 generated frames, denoising each latent frame in a small number of steps (2–4).

Input

• Input Type(s): Image (camera frame), a stream of 44-dimensional numerical action vectors
• Input Format(s): Red, Green, Blue (RGB) frame, numeric vector
• Input Parameters: Image: Two-Dimensional (2D) image frame; Vector: stream of forty-four-dimensional (44D) action vectors, consumed in chunks (each 12-frame output block consumes 12 action steps) Other Properties Related to Input:
• Recommended resolution: The model operates at 288 × 512 (height × width); a 16:9 aspect ratio is recommended. Input frames are automatically resized.
• Control rate: Effective 10 Hz for dVRK (30 Hz capture, stride 3).
• Pre-processing: Action values are normalized using precomputed statistics. The dVRK kinematic action is a hybrid relative action as defined here. In intera