Model reference · open weights
wav2vec2-large-xlsr-53-spanish-with-lm is an open-weight audio or speech model from patrickvonplaten. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | patrickvonplaten |
|---|---|
| Type | Audio & music |
| Task | Speech→text |
| Runs with | transformers |
| Released | 2022-03-02 |
| Popularity | 2k downloads / month |
| Licence | Open weights |
About
This is a model copy of Wav2Vec2-Large-XLSR-53-Spanish that has language model support.
This model card can be seen as a demo for the pyctcdecode integration with Transformers led by this PR. The PR explains in-detail how the integration works.
In a nutshell: This PR adds a new Wav2Vec2WithLMProcessor class as drop-in replacement for Wav2Vec2Processor.
The only change from the existing ASR pipeline will be:
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm"
sample = next(iter(load_dataset("common_voice", "es", split="test", streaming=True)))
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
logits = model(input_values).logits
-prediction_ids = torch.argmax(logits, dim=-1)
-transcription = processor.batch_decode(prediction_ids)
+transcription = processor.batch_decode(logits.numpy()).text
# => 'bien y qué regalo vas a abrir primero'
Improvement
This model has been compared on 512 speech samples from the Spanish Common Voice Test set and gives a nice 20 % performance boost:
The results can be reproduced by running from this model repository:
| Model | WER | CER |
|---|---|---|
| patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm | 8.44% | 2.93% |
| jonatasgrosman/wav2vec2-large-xlsr-53-spanish | 10.20% | 3.24% |
bash run_ngram_wav2vec2.py 1 512
bash run_ngram_wav2vec2.py 0 512
with run_ngram_wav2vec2.py being
https://huggingface.co/patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm/blob/main/run_ngram_wav2vec2.py
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys wav2vec2-large-xlsr-53-spanish-with-lm for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (wav2vec2-large-xlsr-53-spanish-with-lm below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/audio/transcriptions \ -H "Authorization: Bearer $AXFORGE_API_KEY" \ -F model="wav2vec2-large-xlsr-53-spanish-with-lm" -F file=@audio.mp3
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.