TBDub

See TBDub in action · 46-second demo

One voice track, four faces — multilingual lip sync with the two-step Student Model. Watch examples in seven languages across microphone and on-screen text occlusions, large head turns, and rapid head motion. The video opens with the original clips, then shows the generated results. Turn on sound to hear the language changes.

Open the video · Run the Student Model

V1.1 — Inference and deployment update

V1.1 reduces checkpoint download sizes and adds an optional lower-memory mode to the original V1.0 release. It uses the same trained Teacher and Student models; the paper's quality and H20 speed results remain the original evaluation.

• Smaller downloads: each variant needs one complete BF16 DiT checkpoint of about 12.59 GB, plus the shared auxiliary files listed below. Download only the Teacher or Student you plan to use.
• Optional lower GPU memory use: add --cpu-offload if inference runs out of GPU memory. In this mode, inactive models stay in system RAM. Measured process peaks were 19.49 GiB for Teacher and 15.01 GiB for Student at 512×512 and 376 output frames on an RTX PRO 5000 72GB. See the measured workload and tradeoffs below; these are separate from the paper's H20 speed benchmark.

V1.1 release notes · Project update and memory measurements

Why TBDub?

• 🏆 Leading perceptual quality. Highest mean opinion scores in all three dimensions among the open-source methods in our study: 3.85 lip sync, 3.80 visual quality, and 3.78 identity preservation. Student leads the first two; Teacher leads identity. See results.
• ⚡ Fast, two-step generation. 7.13 FPS on a single NVIDIA H20 at 512 × 512, or 13.93× the Teacher's core generation speed. See benchmark.
• 🌍 Multilingual lip sync. Lip sync across languages, with examples including English, Chinese, Japanese, Korean, and Russian. See lip sync and identity preservation in multilingual reconstruction. Watch examples.
• 🛡️ Robust in challenging scenes. Stable lip sync and appearance through large head turns, hands and microphones over the mouth, and rapid motion. Head turns · Occlusions.

MOS: 38 TalkVid clips, 114 ratings per method, on a 0–5 scale. Speed: VAE encode to decode, excluding preprocessing, audio encoding, and file output. Full evaluation protocol.

Watch comparisons · Get the code · Download models

Model Variants

Choose one variant. Each checkpoint contains the complete DiT model and uses the same runtime parameter values as V1.0. Both variants also need the shared VAE, HuBERT, fixed prompt embedding, and Face Landmarker files listed below. For either variant, the complete required model download is about 15.27 GB: 12.59 GB for its DiT checkpoint plus about 2.68 GB of shared auxiliary files. The other variant's checkpoint is unnecessary. Sizes here use decimal GB and exclude Python packages and CUDA libraries.

For standard 30-step inference, download tbdub_teacher.safetensors. Its 12.59 GB DiT download replaces the 22.87 GB required by the V1.0 setup.

Exact size: 12,591,523,040 bytes. Teacher SHA-256: e73483dfde3b960d3d22f85e356c378067ccd3cff81f3b668d92167b8de605d0.

For two-step inference, download tbdub_student.safetensors. Its BF16 file is 12,591,523,048 bytes (about 12.59 GB), compared with the 37.77 GB DiT download documented for V1.0. All 1,245 tensors match the original Student cast to BF16, the dtype already used by inference. The storage change reduces download size; the optional CPU offload described below reduces GPU memory use.

SHA-256: 4e86900f9744b3650bfe58451ae89b3afba758233a3edf5947516268c1ea11e6.

The manifest lists each variant's single checkpoint, its checksum, and the shared auxiliary dependencies.

Installation and Inference

The inference code and complete installation instructions are available in the GitHub repository:

• GitHub: https://github.com/TaoLiveAIGC/TBDub

Clone the repository:

git clone https://github.com/TaoLiveAIGC/TBDub.git
cd TBDub

Use Linux with an NVIDIA GPU, Python 3.10 (validated environment), and ffmpeg on PATH. Install a CUDA-compatible PyTorch/torchvision build in your environment, then install the project dependencies:

pip install -r requirements.txt

Keep only the pinned opencv-contrib-python package; do not install opencv-python alongside it because both provide cv2.

Download the configuration manifest together with the model variant you plan to use. The current GitHub inference entry point reads checkpoints/config.json, validates its runtime compatibility, and uses it