Model reference · open weights
seamless-m4t-medium is an open-weight audio or speech model from facebook, listed in the AxForge catalogue. AxForge can bring it up on EU-owned hardware for you on request — with the licence handled where one is required.
About
SeamlessM4T Medium SeamlessM4T is a collection of models designed to provide high quality translation, allowing people from different linguistic communities to communicate effortlessly through speech and text. This repository hosts 🤗 Hugging Face's implementation of SeamlessM4T. You can find the original weights, as well as a guide on how to run them in the original hub repositories (large and medium checkpoints). 🌟 SeamlessM4T v2, an improved version of this version with a novel architecture, has been released here. This new model improves over SeamlessM4T v1 in quality as well as inference speed in speech generation tasks. SeamlessM4T v2 is also supported by 🤗 Transformers, more on it in the model card of this new version or directly in 🤗 Transformers docs. SeamlessM4T Medium covers: - 📥 101 languages for speech input - ⌨️ 196 Languages for text input/output - 🗣️ 35 languages for speech output. This is the "medium" variant of the unified model, which enables multiple tasks without relying on multiple separate models: - Speech-to-speech translation (S2ST) - Speech-to-text translation (S2TT) - Text-to-speech translation (T2ST) - Text-to-text translation (T2TT) - Automatic speech recognition (ASR) You can perform all the above tasks from one single model, SeamlessM4TModel, but each task also has its own dedicated sub-model. 🤗 Usage First, load the processor and a checkpoint of the model: You can seamlessly use this model on text or on audio, to generated either translated text or translated audio. Here is how to use the processor to process text and audio: Speech SeamlessM4TModel can seamlessly generate text or speech with few or no changes. Let's target Russian voice translation: With basically the same code, I've translated English text and Arabic speech to Russian speech samples. Text Similarly, you can generate translated text from audio files or from text with the same model. You only have to pass generatespeech=False to SeamlessM4TModel.generate. This time, let's translate to French. Tips 1. Use dedicated models SeamlessM4TModel is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without a
Summarised from the published model card. Read the full card on the HuggingFace links below.
Specifications
| Maker | |
|---|---|
| Type | Audio & music |
| Variants | 1 |
| Runs with | transformers |
| Released | 2023-08-28 |
| Popularity | 115k downloads / month |
| Likes | 32 |
| Licence | Commercial licence needed |
How it works
Variants
Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.
| Variant | Params | Precision | VRAM | Fits 16 GB | Weights |
|---|---|---|---|---|---|
| hf-seamless-m4t-medium | — | BF16 | — | — | Weights ↗ |
Using it via the API
Once AxForge deploys seamless-m4t-medium for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (seamless-m4t-medium below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/audio/transcriptions \ -H "Authorization: Bearer $AXFORGE_API_KEY" \ -F model="seamless-m4t-medium" -F file=@audio.mp3
Details
Tags
Licence
The weights are open but cc-by-nc-4.0 needs a commercial agreement for business use. AxForge can arrange that licence and host the model for you — you pay AxForge, we settle with the model’s maker. Ask us for a quote. Read the licence ↗