Model reference · open weights
Lumina-Next-SFT is an open-weight image model from Alpha-VLLM. Lumina-Next-SFT-diffusers (BF16) weighs 8.9 GB; the smallest configuration that runs it is RTX 3060 12 GB.
Summary of the Alpha-VLLM/Lumina-Next-SFT-diffusers model card, 2026-10-01
What it is
| Released by | Alpha-VLLM |
|---|---|
| Released | 2024-06-20 |
| Parameters | 1.8B |
| VRAM | 8.9 GB for the weights |
What it runs on
| Card | One image | Memory |
|---|---|---|
| RTX 3060 12 GB | tight (encoders offloaded) | 11.6 GB |
| RTX 4060 Ti 16 GB | tight | 15.4 GB |
| RTX 3090 24 GB | fits | 23.4 GB |
| RTX 4090 24 GB | fits | 23.4 GB |
| RTX 5090 32 GB | fits | 31.0 GB |
| L40S 48 GB | fits | 44.0 GB |
| A100 80 GB | fits | 78.2 GB |
| H100 80 GB | fits | 78.1 GB |
| RTX PRO 6000 Blackwell 96 GB | fits | 93.8 GB |
| DGX Spark (GB10) 128 GB unified | fits | 107 GB |
| H200 141 GB | fits | 138 GB |
| B200 180 GB | fits | 176 GB |
Builds
| Build | Params | Precision | Weights | Smallest setup |
|---|---|---|---|---|
| Lumina-Next-SFT-diffusers (above) ↗ | 1.8B | BF16 | 8.9 GB | RTX 3060 12 GB |
| Lumina-Next-SFT ↗ | — | BF16 | 13.0 GB | RTX 4060 Ti 16 GB |
From the model card
The Lumina-Next-SFT is a Next-DiT model containing 2B parameters and utilizes Gemma-2B as the text encoder, enhanced through high-quality supervised fine-tuning (SFT).
Our generative model has Next-DiT as the backbone, the text encoder is the Gemma 2B model, and the VAE uses a version of sdxl fine-tuned by stabilityai.
[2024-07-08] 🎉🎉🎉 Lumina-Next is now supported in the diffusers! Thanks to @yiyixuxu and @sayakpaul!
[2024-06-08] 🎉🎉🎉 We have released the Lumina-Next-SFT model.
[2024-05-28] We updated the Lumina-Next-T2I model to support 2K Resolution image generation.
[2024-05-16] We have converted the .pth weights to .safetensors weights. Please pull the latest code to use demo.py for inference.
[2024-05-12] We release the next version of Lumina-T2I, called Lumina-Next-T2I for faster and lower memory usage image generation model.
More checkpoints of our model will be released soon~
| Resolution | Next-DiT Parameter | Text Encoder | Prediction | Download URL |
|---|---|---|---|---|
| 1024 | 2B | Gemma-2B | Rectified Flow | hugging face |
Note: You may want to adjust the CUDA version according to your driver version.
conda create -n Lumina_T2X -y
conda activate Lumina_T2X
conda install python=3.11 pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y
pip install diffusers huggingface_hub
flash-attnpip install flash-attn --no-build-isolation
⭐⭐ (Recommended) you can use huggingface_cli to download our model:
huggingface-cli download --resume-download Alpha-VLLM/Lumina-Next-SFT-diffusers --local-dir /path/to/ckpt
from diffusers import LuminaText2ImgPipeline
import torch
pipeline = LuminaText2ImgPipeline.from_pretrained("/path/to/ckpt/Lumina-Next-SFT-diffusers", torch_dtype=torch.bfloat16).to("cuda")
# or you can download the model using code directly
# pipeline = LuminaText2ImgPipeline.from_pretrained("Alpha-VLLM/Lumina-Next-SFT-diffusers", torch_dtype=torch.bfloat16).to("cuda")
image = pipeline(prompt="Upper body of a young woman in a Victorian-era outfit with brass goggles and leather straps. "
"Background shows an industrial revolution cityscape with smoky skies and tall, metal structures").images[0]
Quoted from the model card on Hugging Face. The full card is behind the Hugging Face link above.
How it works
Running it yourself
Rent a machine by the hour. Runs as it is with diffusers, on the machine, in Python.
# on your rented machine: pip install diffusers transformers accelerate
import torch
from diffusers import DiffusionPipeline
pipe = DiffusionPipeline.from_pretrained("Alpha-VLLM/Lumina-Next-SFT-diffusers", torch_dtype=torch.bfloat16).to("cuda")
image = pipe(prompt="a red bicycle on a cobbled street").images[0]
image.save("/workspace/out.png")