Model reference · open weights
bloomz-petals is an open-weight language model from bigscience. bloomz-petals (BF16) weighs 7.2 GB; the smallest configuration that runs it is L40S 48 GB.
Summary of the bigscience/bloomz-petals model card, 2026-10-01
What it is
| Released by | bigscience |
|---|---|
| Released | 2023-01-16 |
| VRAM | 7.2 GB for the weights |
What it runs on
| Card | Requests at once | Context max | Memory | |
|---|---|---|---|---|
| 8K each | 32K each | |||
| RTX 3060 12 GB | — | — | 755 | 11.6 GB |
| RTX 4060 Ti 16 GB | — | — | 1K | 15.4 GB |
| RTX 3090 24 GB | — | — | 3K | 23.4 GB |
| RTX 4090 24 GB | — | — | 3K | 23.4 GB |
| RTX 5090 32 GB | — | — | 5K | 31.0 GB |
| L40S 48 GB | 1 | — | 8K | 44.0 GB |
| A100 80 GB | 2 | — | 16K | 78.2 GB |
| H100 80 GB | 1 | — | 15K | 78.1 GB |
| RTX PRO 6000 Blackwell 96 GB | 2 | — | 19K | 93.8 GB |
| DGX Spark (GB10) 128 GB unified | 2 | — | 22K | 107 GB |
| H200 141 GB | 3 | — | 29K | 138 GB |
| B200 180 GB | 4 | 1 | 38K | 176 GB |
| 2× RTX 4090 24 GB tensor parallel | 1 | — | 8K | 23.4 GB a card |
| 2× RTX 3090 24 GB tensor parallel | 1 | — | 8K | 23.4 GB a card |
| 2× RTX 5090 32 GB tensor parallel | 1 | — | 12K | 31.0 GB a card |
| 2× L40S 48 GB tensor parallel | 2 | — | 18K | 44.0 GB a card |
| 4× H100 80 GB tensor parallel | 8 | 2 | 66K | 78.1 GB a card |
| 4× A100 80 GB tensor parallel | 9 | 2 | 72K | 78.2 GB a card |
| 2× B200 180 GB tensor parallel | 9 | 2 | 78K | 176 GB a card |
| 4× RTX PRO 6000 Blackwell 96 GB tensor parallel | 10 | 2 | 81K | 93.8 GB a card |
| 4× H200 141 GB tensor parallel | 15 | 3 | 124K | 138 GB a card |
| Requests at once | 8K tokens each | 32K tokens each |
|---|---|---|
| 1 | 41.5 GB | 140 GB |
| 5 | 173 GB | 666 GB |
| 8 | 272 GB | 1061 GB |
| 16 | 535 GB | 2113 GB |
| 32 | 1061 GB | 4218 GB |
| 64 | 2113 GB | 8427 GB |
One card, with vLLM's small-card settings.
From the model card
This model is a version of bigscience/bloomz post-processed to be run at home using the Petals swarm.
Please check out:
We provide minimal code examples below.
from petals import DistributedBloomForCausalLM
model = DistributedBloomForCausalLM.from_pretrained("bigscience/bloomz-petals")
# Embeddings & prompts are on your device, BLOOM blocks are distributed across the Internet
inputs = tokenizer("A cat sat", return_tensors="pt")["input_ids"]
outputs = model.generate(inputs, max_new_tokens=5)
print(tokenizer.decode(outputs[0])) # A cat sat on a mat...
python -m petals.cli.run_server bigscience/bloomz-petals
Quoted from the model card on Hugging Face. The full card is behind the Hugging Face link above.