Video-As-Prompt-CogVideoX

🔥 News

• Oct 24, 2025: 📖 We release the first unified semantic video generation model, Video-As-Prompt (VAP)!
• Oct 24, 2025: 🤗 We release the VAP-Data, the largest semantic-controlled video generation datasets with more than $100K$ samples!
• Oct 24, 2025: 👋 We present the technical report of Video-As-Prompt, please check out the details and spark some discussion!

🖌️ Video-As-Prompt

Core idea: Given a reference video with wanted semantics as a video prompt, Video-As-Prompt animate a reference image with the same semantics as the reference video. controls autoplay playsinline muted loop src="https://github.com/user-attachments/assets/2e440927-5b16-4761-ad1f-46ac93de2d8e" width="60%"

Your browser does not support HTML5 video. Here is a link to the video instead.

Welcome to see our project page for more interesting results!

🎁 Models Zoo

To demonstrate cross-architecture generality, Video-As-Prompt provides two variants, each with distinct trade-offs:

• CogVideoX-I2V-5B

  • Strengths: Fewer backbone parameters let us train more steps under limited resources, yielding strong stability on most semantic conditions.
  • Limitations: Due to backbone ability limitation, it is weaker on human-centric generation and on concepts underrepresented in pretraining (e.g., ladudu, Squid Game, Minecraft).

• Wan2.1-I2V-14B

  • Strengths: Strong performance on human actions and novel concepts, thanks to a more capable base model.
  • Limitations: Larger model size reduced feasible training steps given our resources, lowering stability on some semantic conditions.

👏👏👏 Contributions and further optimization from the community are welcome.

Please download the pre-trained video DiTs and our corresponding Video-As-Prompt models, and structure them as follows

ckpts/
  ├── Video-As-Prompt-CogVideoX-5B/
      ├── scheduler
      ├── vae
      ├── transformer
      ├── ...
  ├── Video-As-Prompt-Wan2.1-14B/
      ├── scheduler
      ├── vae
      ├── transformer
      ├── ...

🤗 Get Started with Video-As-Prompt

Video-As-Prompt supports Macos, Windows, Linux. You may follow the next steps to use Video-As-Prompt via:

Install Requirements

We test our model with Python 3.10 and PyTorch 2.7.1+cu124.

conda create -n video_as_prompt python=3.10 -y
conda activate video_as_prompt
pip install -r requirements.txt
pip install -e ./diffusers
conda install -c conda-forge ffmpeg -y

Data

We have published the VAP-Data dataset used in our paper on VAP-Data. Please download it and put it in the data folder. The structure should look like:

data/
  ├── VAP-Data/
  │   ├── vfx_videos/
  │   ├── vfx_videos_hq/
  │   ├── vfx_videos_hq_camera/
  │   ├── benchmark/benchmark.csv
  │   ├── vap_data.csv

Code Usage

We mainly implement our code based on diffusers and finetrainers for their modular design.

Minimal Demo

Below is a minimal demo of our CogVideoX-I2V-5B variant. The full code can be found in infer/cog_vap.py. The WAN2.1-I2V-14B variant is similar and can be found in infer/wan_vap.py.

import torch
from diffusers import (
    AutoencoderKLCogVideoX,
    CogVideoXImageToVideoMOTPipeline,
    CogVideoXTransformer3DMOTModel,
)
from diffusers.utils import export_to_video, load_video
from PIL import Image

vae = AutoencoderKLCogVideoX.from_pretrained("ByteDance/Video-As-Prompt-CogVideoX-5B", subfolder="vae", torch_dtype=torch.bfloat16)
transformer = CogVideoXTransformer3DMOTModel.from_pretrained("ByteDance/Video-As-Prompt-CogVideoX-5B", torch_dtype=torch.bfloat16)
pipe = CogVideoXImageToVideoMOTPipeline.from_pretrained(
    "ByteDance/Video-As-Prompt-CogVideoX-5B", vae=vae, transformer=transformer, torch_dtype=torch.bfloat16
).to("cuda")

ref_video = load_video("assets/videos/demo/object-725.mp4")
image = Image.open("assets/images/demo/animal-2.jpg").convert("RGB")
idx = torch.linspace(0, len(ref_video) - 1, 49).long().tolist()
ref_frames = [ref_video[i] for i in idx]

output_frames = pipe(
    image=image,
    ref_videos=[ref_frames],
    prompt="A chestnut-colored horse stands on a grassy hill against a backdrop of distant, snow-dusted mountains. The horse begins to inflate, its defined, muscular body swelling and rounding into a smooth, balloon-like form while retaining its rich, brown hide color. Without changing its orientation, the now-buoyant horse lifts silently from the ground. It begins a steady vertical ascent, rising straight up and eventually floating out of the top of the frame. The camera remains completely static throughout the entire sequence, holding a fixed shot on the landscape as the horse transforms and departs, ensuring the verdant hill and mountain range in the background stay perfectly still.",
    prompt_mot_ref=[
      "A hand holds up a single beige sneaker decorated with gold calligraphy and floral illustrations, with small green plants tucked inside. The snea