Model reference · open weights
TinySolar-4k-code is an open-weight language model from upstage. TinySolar-248m-4k-code-instruct (BF16) weighs 496 MB; the smallest configuration that runs it is RTX 3060 12 GB.
What it is
| Released by | upstage |
|---|---|
| Released | 2024-04-15 |
| Parameters | 248M |
| VRAM | 496 MB for the weights |
What it runs on
| Card | Requests at once | Context max | Memory | |
|---|---|---|---|---|
| 8K each | 32K each | |||
| RTX 3060 12 GB | 105 | 26 | all 32K | 11.6 GB |
| RTX 4060 Ti 16 GB | 143 | 35 | all 32K | 15.4 GB |
| RTX 3090 24 GB | 222 | 55 | all 32K | 23.4 GB |
| RTX 4090 24 GB | 221 | 55 | all 32K | 23.4 GB |
| RTX 5090 32 GB | 298 | 74 | all 32K | 31.0 GB |
| L40S 48 GB | 426 | 106 | all 32K | 44.0 GB |
| A100 80 GB | 766 | 191 | all 32K | 78.2 GB |
| H100 80 GB | 739 | 184 | all 32K | 78.1 GB |
| RTX PRO 6000 Blackwell 96 GB | 895 | 223 | all 32K | 93.8 GB |
| DGX Spark (GB10) 128 GB unified | 1000+ | 257 | all 32K | 107 GB |
| H200 141 GB | 1000+ | 333 | all 32K | 138 GB |
| B200 180 GB | 1000+ | 428 | all 32K | 176 GB |
| Requests at once | 8K tokens each | 32K tokens each |
|---|---|---|
| 1 | 1.1 GB | 1.4 GB |
| 5 | 1.5 GB | 3.0 GB |
| 8 | 1.8 GB | 4.2 GB |
| 16 | 2.6 GB | 7.5 GB |
| 32 | 4.2 GB | 13.9 GB |
| 64 | 7.5 GB | 26.8 GB |
One card, with vLLM's small-card settings.
From the model card
Quoted from the model card on Hugging Face. The full card is behind the Hugging Face link above.