Cosmos-H-Surgical

Model Overview

Cosmos-H-Surgical v0.3.0 is a unified surgical video world model fine-tuned from NVIDIA Cosmos3-Nano. One checkpoint supports two workflows:

• Video prediction: generate the next 92 surgical video frames from a starting image and a structured text description.
• Simulation-to-real transfer: convert a blur, depth, segmentation, or edge control video into a photorealistic surgical video.

Prediction requires a starting image. Each transfer input specification activates exactly one control type. The four controls may be evaluated for the same source case as separate specifications.

This model is ready for commercial and non-commercial use.

The earlier Cosmos 2.5 release remains available from the cosmos-2.5 branch.

License

The v0.3.0 model weights and accompanying Cosmos-H-Surgical source code are provided under OpenMDW-1.1. The official license source is available from the OpenMDW repository. Third-party software and upstream model components remain subject to their own license terms.

Intended Use

Expected users include medical researchers, surgical robotics developers, AI developers, and healthcare institutions working on:

• Synthetic surgical video generation for downstream robotics research
• Simulation-to-real transfer for surgical scenes
• Physical AI development and evaluation

This model is not intended for clinical diagnosis or autonomous clinical decision-making. Validate model behavior with use-case-specific data before deployment. Users must have appropriate rights and permissions for all input images and videos.

Model Architecture

Inputs

Inference requires an outer JSON or JSONL specification. Its prompt field is a serialized structured JSON object describing subjects, scene, actions, and temporal evolution. See the inference documentation for the complete schema.

Prediction

• One starting JPG, PNG, JPEG, or WebP image
• One structured text prompt
• model_mode: "image2video"

Transfer

• One control modality per JSON specification
• Blur and edge controls use a source RGB video
• Depth and segmentation controls use their corresponding control videos
• model_mode: "video2video"

The validated release settings are 480P, 16 FPS, 93 frames, guidance 6 and shift 5 for prediction, and guidance 1, control guidance 1 and shift 5 for transfer.

Outputs

The model produces an MP4 surgical video at 832 x 480, 16 FPS, and 93 frames. Prediction includes the starting frame followed by 92 generated frames. Transfer preserves the temporal and spatial structure of its control input while synthesizing a photorealistic appearance.

Installation

Clone the Cosmos-H-Surgical v0.3.0 source release and install one CUDA group:

# CUDA 13, recommended
uv sync --group cu130

# CUDA 12.8
uv sync --group cu128

source .venv/bin/activate

Inference

The Cosmos-H-Surgical checkpoint is selected automatically when --checkpoint-path is omitted:

torchrun --nproc-per-node=8 \
  -m cosmos_h_surgical infer \
  --parallelism-preset=latency \
  --dp-shard-size=1 \
  -i inputs/predict/surgical_predict.jsonl \
  -o outputs/cosmos-h-surgical/predict \
  --no-guardrails \
  --seed=0

The explicit equivalent is:

--checkpoint-path Cosmos-H-Surgical

For transfer, pass the prepared specifications together so the model is loaded once:

torchrun --nproc-per-node=8 \
  -m cosmos_h_surgical infer \
  --parallelism-preset=latency \
  --dp-shard-size=1 \
  --no-use-torch-compile \
  -i "inputs/transfer/specs/*.json" \
  -o outputs/cosmos-h-surgical/transfer \
  --no-guardrails \
  --seed=0

The first inference downloads the checkpoint and tokenizer dependencies into the Hugging Face cache; later runs reuse the cache.

Training and Evaluation Data

Dataset Overview

Cosmos-H-Surgical was trained using GraSP real surgical video and synthetic surgical video data. The synthetic collection contains 12,600 laparoscopic cholecystectomy videos generated through the Cosmos-H-Surgical transfer pipeline. GraSP contributes 15,043 robot-assisted radical prostatectomy videos. The combined collection uses a 90% training and 10% validation split.

The training collection contains approximately 24,879 videos and the held-out validation collection approximately 2,764 videos. Total video duration is less than 10,000 hours.

Hardware and Software

• Linux
• BF16 inference
• NVIDIA Hopper was used for release validation
• Eight H100 80 GB GPUs with the latency parallelism preset
• CUDA 13 recommended; CUDA 12.8 supported by the release environment

Other GPU architectures, precisions, and parallelism settings require separate validation.

References

1. NVIDIA Cosmos: World Foundation Models for Physical AI
2. NVIDIA Cosmos3-Nano
3. Cosmos-H-Surgical source repository

Please report security vulnerabilities through the [NVIDIA Product Security process](https://www.nvidia.com/en-us/support/submit-