Cosmos-1.0-Diffusion-Text2World

Model Developer: NVIDIA

Model Versions

In Cosmos 1.0 release, the Cosmos Diffusion WFM family includes the following models:

• Cosmos-1.0-Diffusion-7B-Text2World
  • Given a text description, predict an output video of 121 frames.
• Cosmos-1.0-Diffusion-14B-Text2World
  • Given a text description, predict an output video of 121 frames.
• Cosmos-1.0-Diffusion-7B-Video2World
  • Given a text description and an image as the first frame, predict the future 120 frames.
• Cosmos-1.0-Diffusion-14B-Video2World
  • Given a text description and an image as the first frame, predict the future 120 frames.

License:

This model is released under the NVIDIA Open Model License. For a custom license, please contact cosmos-license@nvidia.com.

Under the NVIDIA Open Model License, NVIDIA confirms:

• Models are commercially usable.
• You are free to create and distribute Derivative Models.
• NVIDIA does not claim ownership to any outputs generated using the Models or Derivative Models.

Important Note: If you bypass, disable, reduce the efficacy of, or circumvent any technical limitation, safety guardrail or associated safety guardrail hyperparameter, encryption, security, digital rights management, or authentication mechanism contained in the Model, your rights under NVIDIA Open Model License Agreement will automatically terminate.

• Cosmos-1.0-Guardrail is the safety guardrail for this model.

Model Architecture:

Cosmos-1.0-Diffusion-7B-Text2World is a diffusion transformer model designed for video denoising in the latent space. The network is composed of interleaved self-attention, cross-attention and feedforward layers as its building blocks. The cross-attention layers allow the model to condition on input text throughout the denoising process. Before each layers, adaptive layer normalization is applied to embed the time information for denoising. When image or video is provided as input, their latent frames are concatenated with the generated frames along the temporal dimension. Augment noise is added to conditional latent frames to bridge the training and inference gap.

Input/Output Specifications

• Input

  • Input Type(s): Text
  • Input Format(s): String
  • Input Parameters: One-dimensional (1D)
  • Other Properties Related to Input:
    • The input string should contain fewer than 300 words and should provide descriptive content for world generation, such as a scene description, key objects or characters, background, and any specific actions or motions to be depicted within the 5-second duration.

• Output

  • Output Type(s): Video
  • Output Format(s): mp4
  • Output Parameters: Three-dimensional (3D)
  • Other Properties Related to Output: By default, the generated video is a 5-second clip with a resolution of 1280x704 pixels and a frame rate of 24 frames per second (fps). The video content visualizes the input text description as a short animated scene, capturing key elements within the specified time constraints. Aspect ratios and resolutions are configurable, with options including 1:1 (960x960 pixels), 4:3 (960x704 pixels), 3:4 (704x960 pixels), 16:9 (1280x704 pixels), and 9:16 (704x1280 pixels). The frame rate is also adjustable within a range of 12 to 40 fps.

Software Integration

Runtime Engine(s):

• Cosmos
• Diffusers

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Blackwell
• NVIDIA Hopper
• NVIDIA Ampere

Note: We have only tested inference with BF16 precision.

Operating System(s):

• Linux (We have not tested on other operating systems.)

Usage

• See Cosmos for details.

Cosmos can also be used with Diffusers!

import torch
from diffusers import CosmosTextToWorldPipeline
from diffusers.utils import export_to_video

model_id = "nvidia/Cosmos-1.0-Diffusion-7B-Text2World"
pipe = CosmosTextToWorldPipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16)
pipe.to("cuda")

prompt = "A sleek, humanoid robot stands in a vast warehouse filled with neatly stacked cardboard boxes on industrial shelves. The robot's metallic body gleams under the bright, even lighting, highlighting its futuristic design and intricate joints. A glowing blue light emanates from its chest, adding a touch of advanced technology. The background is dominated by rows of boxes, suggesting a highly organized storage system. The floor is lined with wooden pallets, enhancing the industrial setting. The camera remains static, ca