Model reference · open weights
BioMedLM is an open-weight language model from stanford-crfm. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | stanford-crfm |
|---|---|
| Type | Language models |
| Task | Text gen |
| Context | 1k tokens |
| Runs with | transformers |
| Released | 2022-12-14 |
| Popularity | 1k downloads / month |
| Licence | Commercial licence needed |
About
Note: This model was previously known as PubMedGPT 2.7B, but we have changed it due to a request from the NIH which holds the trademark for "PubMed".
Paper: BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
BioMedLM 2.7B is new language model trained exclusively on biomedical abstracts and papers from The Pile. This GPT-style model can achieve strong results on a variety of biomedical NLP tasks, including a new state of the art performance of 50.3% accuracy on the MedQA biomedical question answering task.
As an autoregressive language model, BioMedLM 2.7B is also capable of natural language generation. However, we have only begun to explore the generation capabilities and limitations of this model, and we emphasize that this model’s generation capabilities are for research purposes only and not suitable for production. In releasing this model, we hope to advance both the development of biomedical NLP applications and best practices for responsibly training and utilizing domain-specific language models; issues of reliability, truthfulness, and explainability are top of mind for us.
This model was a joint collaboration of Stanford CRFM and MosaicML.
BioMedLM 2.7B is new language model trained exclusively on biomedical abstracts and papers from The Pile. This GPT-style model can achieve strong results on a variety of biomedical NLP tasks, including a new state of the art performance of 50.3% accuracy on the MedQA biomedical question answering task.
As an autoregressive language model, BioMedLM 2.7B is also capable of natural language generation. However, we have only begun to explore the generation capabilities and limitations of this model, and we emphasize that this model’s generation capabilities are for research purposes only and not suitable for production. In releasing this model, we hope to advance both the development of biomedical NLP applications and best practices for responsibly training and utilizing domain-specific language models; issues of reliability, truthfulness, and explainability are top of mind for us.
This model was a joint collaboration of Stanford CRFM and MosaicML.
This model is licensed under the terms of BigScience Open RAIL-M license used for BLOOM. Please note that, among other restrictions, this license forbids use of the model (or derivatives thereof) "To provide medical advice and medical results interpretation." If you are concerned that your use case would follow under the "letter" of this restriction, but not the "spirit," you can contact us to discuss.
It is possible to use this model to generate text, which is useful for experimentation and understanding its capabilities. It should not be directly used for production or work that may directly impact people.
The main way we have used this model is finetuning for downstream question answering tasks, and we recommend using this model that way.
We do not recommend using this model for natural language generation in a production environment, finetuned or otherwise.
Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
While this model is capable of generating natural language text, we have only begun to explore this capability and its limitations. Understanding these limitations is especially important in a domain like medicine. Therefore, we strongly recommend against using this model in production for natural language generation.
This model was trained on the Pubmed Abstracts and Full Text from The Pile.
The model was trained on MosaicML Cloud, a platform designed for large workloads like LLMs. Using the Composer training library and PyTorch FSDP, it was easy to enable multi-node training across 128 A100-40GB GPUs, and the total run was completed in ~6.25 days. The model was trained with batch size=1024 and sequence length=1024 for 300B tokens using Decoupled AdamW with the following settings:
| lr | 1.6e-4 |
| eps | 1e-8 |
| betas | [0.9, 0.95] |
| weight decay | 1.6e-5 |
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys biomedlm for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (biomedlm below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"biomedlm","messages":[{"role":"user","content":"Hello"}]}'
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.