Model reference · open weights
Lumina-mGPT-512 is an open-weight language model from Alpha-VLLM. Lumina-mGPT-34B-512 (BF16) weighs 68.6 GB; the smallest configuration that runs it is H100 80 GB.
What it is
| Released by | Alpha-VLLM |
|---|---|
| Released | 2024-08-04 |
| Parameters | 7.0B |
| VRAM | 68.6 GB for the weights |
What it runs on
| Card | Requests at once | Context max | Memory |
|---|---|---|---|
| 2K each | |||
| RTX 3060 12 GB … L40S 48 GB 6 smaller cards | — | — | |
| A100 80 GB | 21 | all 2K | 78.2 GB |
| H100 80 GB | 12 | all 2K | 78.1 GB |
| RTX PRO 6000 Blackwell 96 GB | 51 | all 2K | 93.8 GB |
| DGX Spark (GB10) 128 GB unified | 84 | all 2K | 107 GB |
| H200 141 GB | 160 | all 2K | 138 GB |
| B200 180 GB | 252 | all 2K | 176 GB |
| 2× L40S 48 GB tensor parallel | 43 | all 2K | 44.0 GB a card |
| 4× RTX 4090 24 GB tensor parallel | 53 | all 2K | 23.4 GB a card |
| 4× RTX 3090 24 GB tensor parallel | 54 | all 2K | 23.4 GB a card |
| 4× RTX 5090 32 GB tensor parallel | 129 | all 2K | 31.0 GB a card |
| 2× H100 80 GB tensor parallel | 194 | all 2K | 78.1 GB a card |
| 2× A100 80 GB tensor parallel | 213 | all 2K | 78.2 GB a card |
| Requests at once | 2K tokens each |
|---|---|
| 1 | 69.8 GB |
| 5 | 71.4 GB |
| 8 | 72.6 GB |
| 16 | 75.8 GB |
| 32 | 82.3 GB |
| 64 | 95.2 GB |
One card, with vLLM's small-card settings.
Builds
| Build | Params | Precision | Weights | Smallest setup |
|---|---|---|---|---|
| Lumina-mGPT-7B-512 ↗ | 7.0B | BF16 | 14.0 GB | 2× RTX 3060 12 GB |
| Lumina-mGPT-34B-512 (above) ↗ | 34.3B | BF16 | 68.6 GB | H100 80 GB |
From the model card
Lumina-mGPT is a family of multimodal autoregressive models capable of various vision and language tasks, particularly excelling in generating flexible photorealistic images from text descriptions.
We provide the implementation of Lumina-mGPT, as well as sampling code, in our github repository.
Quoted from the model card on Hugging Face. The full card is behind the Hugging Face link above.