Model reference · open weights
kotoba-whisper-bilingual is an open-weight audio or speech model from kotoba-tech. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | kotoba-tech |
|---|---|
| Type | Audio & music |
| Task | Speech→text |
| Parameters (lead) | 756M |
| Runs with | transformers |
| Released | 2024-09-27 |
| Popularity | 8k downloads / month |
| Licence | Open weights |
About
faster-whisper weight, whisper.cpp weight
Kotoba-Whisper-Bilingual is a collection of distilled Whisper models trained for
developed through the collaboration bewteen Asahi Ushio and Kotoba Technologies. Following the original work of distil-whisper (Robust Knowledge Distillation via Large-Scale Pseudo Labelling), we employ OpenAI's Whisper large-v3 as the teacher model for Japanese and English ASR, while we translate the transcription into English and Japanese by external LLM to obtain training dataset for speech-to-text translation. We employ ReazonSpeech for Japanese ASR and Japanese speech to English text translation, and Multilingual LibriSpeech for English ASR and English speech to Japanese text translation. Kotoba-whisper-bilingual's loss objective consists of cross-entropy on both of ASR and translation tasks, while KL divergence loss only for ASR task. The student model consists the full encoder of the teacher large-v3 model and the decoder with two layers initialized from the first and last layer of the large-v3 model.
As kotoba-whisper uses the same architecture as distil-whisper/distil-large-v3, it inherits the benefit of the improved latency compared to openai/whisper-large-v3 (6.3x faster than large-v3, see the table below taken from distil-whisper/distil-large-v3).
We compare our kotoba-whisper-bilingual with OpenAI whisper models, kotoba-whisper models, and cascaded models for translation. Worth noting that kotoba-whisper-bilingual is the only model that can do Japanese and English ASR and speech-to-text translation between Japanese and English, as OpenAI whisper is not trained for English to Japanese speech-to-text translation, and other models are specific to the Task (eg. kotoba-whisper is Japanese ASR and distil whisper is English ASR only).
| model | CoVoST2 (Ja->En) | Fleurs (Ja->En) |
|---|---|---|
| kotoba-tech/kotoba-whisper-bilingual-v1.0 | 73.9 | 98.7 |
| japanese-asr/ja-cascaded-s2t-translation (facebook/nllb-200-3.3B) | 64.3 | 67.1 |
| japanese-asr/ja-cascaded-s2t-translation (facebook/nllb-200-1.3B) | 65.4 | 68.9 |
| japanese-asr/ja-cascaded-s2t-translation (facebook/nllb-200-distilled-1.3B) | 65.6 | 67.4 |
| japanese-asr/ja-cascaded-s2t-translation (facebook/nllb-200-distilled-600M) | 68.2 | 72.2 |
| openai/whisper-large-v3 | 71 | 86.1 |
| [openai/whisper-large-v2] |
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys kotoba-whisper-bilingual for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (kotoba-whisper-bilingual below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/audio/transcriptions \ -H "Authorization: Bearer $AXFORGE_API_KEY" \ -F model="kotoba-whisper-bilingual" -F file=@audio.mp3
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.