tara

Highlights

• State-of-the-art Vistaar Hindi aggregate: 12.06 WER mean over the 7 Vistaar sets, ahead of Sarvam Saaras-v3 (12.32), with wins on Kathbath, GramVaani, IndicTTS and CommonVoice-hi.
• Native code-switching: 8.37 WER on Code-Switch FLEURS (CS-FLEURS) Hindi–English read code-switch via Tara's mixed-code mode, competitive with the best commercial systems.
• Robust across domains: read speech, noisy speech, telephony (GramVaani 21.03 vs Sarvam 23.00), spontaneous conversation (IndicVoices), and accented adult/child speech (HiACC).
• Bilingual: retains strong English (6.68 WER CommonVoice-en, 4.55 FLEURS-en).
• Standard tooling: loads with 🤗 Transformers exactly like openai/whisper-large-v3.

Usage

import librosa
import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration

repo = "Trelis/tara"
processor = WhisperProcessor.from_pretrained(repo)
model = WhisperForConditionalGeneration.from_pretrained(
    repo, torch_dtype=torch.bfloat16).to("cuda")

tk = processor.tokenizer
hi, en, mc = (tk.convert_tokens_to_ids(t) for t in ("", "", ""))
trn, nts = (tk.convert_tokens_to_ids(t) for t in ("", ""))

audio_16k, _ = librosa.load("clip.wav", sr=16000, mono=True)
feats = processor(audio_16k, sampling_rate=16000,
                  return_tensors="pt").input_features.to("cuda", torch.bfloat16)

# Example 1: pure Hindi
out = model.generate(input_features=feats,
                     forced_decoder_ids=[(1, hi), (2, trn), (3, nts)],
                     max_new_tokens=444)
print(tk.decode(out[0], skip_special_tokens=True))

# Example 2: Hindi-English mixed-code, inject  right after the language token.
# Language auto-detection also works: generate one step unforced and the FIRST generated
# token is the language token; then inject  after it and continue.
out = model.generate(input_features=feats,
                     forced_decoder_ids=[(1, hi), (2, mc), (3, trn), (4, nts)],
                     max_new_tokens=444)
print(tk.decode(out[0], skip_special_tokens=True))

The mixed-code mode (the `` prefix above) conditions generation only: on pure-Hindi audio it neither degrades accuracy nor forces transliteration; on mixed-code audio it renders English words in Latin script.

Evaluation

Evaluation code, the exact text normalizer, and Tara's per-utterance predictions for every benchmark below are published at TrelisResearch/tara, so all numbers can be reproduced or re-scored under alternative normalizers.

Protocol. All numbers are corpus WER after light text normalization* (Unicode NFC plus punctuation removal; nukta and all vowel and nasal marks preserved). All systems are scored on clips ≤ 30 s with identical references. Commercial-system results are measured by us under the same protocol; they are not vendor-reported figures.

Vistaar Hindi benchmark (WER ↓)

IndicVoices-500 is a 500-sample spontaneous-speech control from the IndicVoices validation split; it is not part of the Vistaar mean.

Code-switching (Hinglish) benchmarks (WER ↓)

Tara and Sarvam are measured in their code-mixed modes.

English (WER ↓)

Scored with the standard Whisper English normalizer.

* Normalization: unicodedata.normalize("NFC"), lowercasing, then removal of punctuation and symbols (। , . ? ! " : ; - – — “ ” ( ) [ ] / ~ % ₹ $ …), invisible formatting characters (zero-width joiner/space)