Model reference · open weights

v0-llama2-100k

LLMs delphi-suite Text gen 1 build Open weights 54k dl/mo

v0-llama2-100k is an open-weight language model from delphi-suite. v0-llama2-100k (BF16) weighs 1 MB; the smallest configuration that runs it is RTX 3060 12 GB.

Hugging Face ↗

What it is

Released by	delphi-suite
Released	2024-01-19
VRAM	1 MB for the weights

What it runs on

Memory and cards for v0-llama2-100k (BF16)

1 MBweights, file size

0 MBcache per 1K tokens

436 MBruntime overhead, at least

512 tokenscontext max

Card	Requests at once	Context max	Memory
Card	512 each	Context max	Memory
RTX 3060 12 GB	1000+	all 512	11.6 GB
RTX 4060 Ti 16 GB	1000+	all 512	15.4 GB
RTX 3090 24 GB	1000+	all 512	23.4 GB
RTX 4090 24 GB	1000+	all 512	23.4 GB
RTX 5090 32 GB	1000+	all 512	31.0 GB
L40S 48 GB	1000+	all 512	44.0 GB
A100 80 GB	1000+	all 512	78.2 GB
H100 80 GB	1000+	all 512	78.1 GB
RTX PRO 6000 Blackwell 96 GB	1000+	all 512	93.8 GB
DGX Spark (GB10) 128 GB unified	1000+	all 512	107 GB
H200 141 GB	1000+	all 512	138 GB
B200 180 GB	1000+	all 512	176 GB

Memory needed at each load

Requests at once	512 tokens each
1	437 MB
5	438 MB
8	438 MB
16	440 MB
32	443 MB
64	449 MB

One card, with vLLM's small-card settings.

Details

Type	Language models
Task	Text gen
Context	512 tokens
Runs with	transformers
Popularity	54k downloads / month
Weights	1 MB, v0-llama2-100k (BF16, file size)