TrorYongASR-tiny

Model Details

Model Description

TrorYongASR is an Encoder-Decoder model for Automatic Speech Recognition (ASR) task. It is inspired by PARSeq and Whisper: the auditory-lingual decoder has only one transformer block.

TrorYongASR has 2 configurations:

Note: The audio array are processed to log-mel spectrogram with 80 mels (the same as Whisper models of the same size)

• Developed by: KHUN Kimang (Ph.D.)
• Shared by: KrorngAI
• Model type: ASR (Automatic Speech Recognition)
• Language(s) (NLP): Khmer and English

Model Sources

• Repository: https://github.com/Kimang18/KrorngAI/tree/main/tror-yong-asr
• Blog Post: https://kimang18.github.io/krorngai-blog/TrorYongASR/
• Demo: https://krorngai-troryongasr-demo.hf.space

Evaluation

The evaluation assesses two capabilities — language detection and transcription — on two datasets (google/fleurs for Khmer and openslr/librispeech_asr for English). All results are from the test split of each dataset, representing the model's generalization ability to unseen data.

Testing Data

Note: Audios longer than 30 seconds are excluded from the evaluation (that is why google/fleurs has 765 examples instead of 771).

Metrics and Results

Language Detection

Language detection measures model’s capability to recognize the spoken language from audio input. Since TrorYongASR currently supports 2 languages, this task becomes binary classification task. Classic metrics are used:

• Precision: Proportion of predicted languages that are correct
• Recall : Proportion of actual language samples correctly identified
• F1-score : Harmonic mean of precision and recall

Results:

Tiny size achieved perfect language detection performance on both datasets, indicating excellent binary classification capability for distinguishing between Khmer and English audio. Small size performs slightly worst by tending to predict English language.

The 100% language detection scores may appear unusually high. This is expected because during pre-training, the model performs permutations on word tokens starting from position 3, while the first three positions (start token, language token, and task token) remain fixed. Since language detection relies on the language token at position 1, and this token is never permuted during pre-training, the model can achieve perfect accuracy on language detection tasks.

Transcription

For transcription task, 3 metrics below are used

• Token Error Rate (TER) : Proportion of incorrectly transcribed tokens
• Character Error Rate (CER) : Proportion of characters that are incorrect
• Word Error Rate (WER) : Proportion of words that are incorrect

Token Error Rate (TER) measures model's capability in predicting the next token given the audio input and the current sequence of tokens. This metric is weaker than Word Error Rate (WER) and Character Error Rate (CER) because it doesn't account for insertions, deletions, substitutions, and autoregression as comprehensively. Token Error Rate is used here because Khmer text lacks word boundaries, making WER and CER calculations challenging without additional preprocessing.

Transcription Results:

Key Observations:

• The tiny model shows strong performance on Engli