Model reference · open weights
evo-1-131k is an open-weight language model from togethercomputer. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Maker | togethercomputer |
|---|---|
| Type | Language models |
| Task | Text gen |
| Parameters (lead) | 6.5B |
| Runs with | transformers |
| Released | 2024-02-20 |
| Popularity | 2k downloads / month |
| Licence | Open weights |
About
We identified and fixed an issue related to a wrong permutation of some projections, which affects generation quality. To use the new model revision, please load as follows:
config = AutoConfig.from_pretrained(model_name, trust_remote_code=True, revision="1.1_fix")
model = AutoModelForCausalLM.from_pretrained(
model_name,
config=config,
trust_remote_code=True,
revision="1.1_fix"
)
Evo is a biological foundation model capable of long-context modeling and design.
Evo uses the StripedHyena architecture to enable modeling of sequences at a single-nucleotide, byte-level resolution with near-linear scaling of compute and memory relative to context length. Evo has 7 billion parameters and is trained on OpenGenome, a prokaryotic whole-genome dataset containing ~300 billion tokens.
We describe Evo in the paper “Sequence modeling and design from molecular to genome scale with Evo”.
As part of our commitment to open science, we release weights of 15 intermediate pretraining checkpoints for phase 1 and phase 2 of pretraining. The checkpoints are available as branches of the corresponding HuggingFace repository.
Evo-1 (Phase 2) is our longer context model in the Evo family, trained at a context length of 131k and tested on generation of sequences of length >650k
We provide the following model checkpoints:
| Checkpoint Name | Description |
|---|---|
evo-1-8k-base | A model pretrained with 8,192 context. We use this model as the base model for molecular-scale finetuning tasks. |
evo-1-131k-base | A model pretrained with 131,072 context using evo-1-8k-base as the base model. We use this model to reason about and generate sequences at the genome scale. |
evo-1-8k-crispr | A model finetuned using evo-1-8k-base as the base model to generate CRISPR-Cas systems. |
evo-1-8k-transposon | A model finetuned using evo-1-8k-base as the base model to generate IS200/IS605 transposons. |
StripedHyena is a deep signal processing, hybrid architecture composed of multi-head attention and gated convolutions arranged in Hyena blocks, improving over decoder-only Transformers.
StripedHyena is designed to leverage the specialization of each of its layer classes, with Hyena layers implementing the bulk of the computation required for sequence processing and attention layers supplementing the ability to perform targeted pattern recall.
Some highlights of the architecture:
Example usage is provided in the standalone repo.
One of the advantages of deep signal processing models is their flexibility. Different parametrizations of convolutions can be used depending on the memory, expressivity and causality requirements of pretraining, finetuning or inference workloads.
The main classes are:
StripedHyena is a mixed precision model. Make sure to keep your poles and residues in float32 precision, especially for longer prompts or training.
To use StripedHyena, you will need to install custom kernels. Please follow the instructions from the standalone repository.
@article{nguyen2024sequence,
author = {Eric Nguyen and Michael Poli and Matthew G. Durrant and Brian Kang and Dhruva Katrekar and David B. Li and Liam J. Bartie and Armin W. Thomas and Samuel H. King and Garyk Brixi and Jeremy Sullivan and Madelena Y. Ng and Ashley Lewis and Aaron Lou and Stefano Ermon and Stephen A. Baccus and Tina Hernandez-Boussard and Christopher Ré and Patrick D. Hsu and Brian L. Hie },
title = {Sequence modeling and design from molecular to genome scale with Evo},
journal = {Science},
volume = {386},
number = {6723},
pages = {eado9336},
year = {2024},
doi = {10.1126/science.ado9336},
URL = {https://www.science.org/doi/abs/10.1126/science.ado9336},
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys evo-1-131k for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (evo-1-131k below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"evo-1-131k","messages":[{"role":"user","content":"Hello"}]}'
Create an account — your API key is available in the console. 5M tokens/month currently included with every new account at launch.