stt_ar_fastconformer_hybrid_large_pc

License

License to use this model is covered by the CC-BY-4.0. By downloading the public and release version of the model, you accept the terms and conditions of the CC-BY-4.0 license.

References

[1] Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition

[2] Google Sentencepiece Tokenizer

[3] NVIDIA NeMo Toolkit

[4] HuggingFace ASR Leaderboard

Model Architecture

FastConformer [1] is an optimized version of the Conformer model with 8x depthwise-separable convolutional downsampling. The model is trained in a multitask setup with hybrid Transducer decoder (RNNT) and Connectionist Temporal Classification (CTC) loss. You may find more information on the details of FastConformer here: Fast-Conformer Model.

Model utilizes a Google Sentencepiece Tokenizer [2] tokenizer with a vocabulary size of 1024.

Input

• Input Type: Audio
• Input Format(s): .wav files
• Other Properties Related to Input: 16000 Hz Mono-channel Audio, Pre-Processing Not Needed

Output

This model provides transcribed speech as a string for a given audio sample.

• Output Type: Text
• Output Format: String
• Output Parameters: One Dimensional (1D)
• Other Properties Related to Output: May Need Inverse Text Normalization; Does Not Handle Special Characters; Outputs text in Arabic without diacritical marks

Limitations

The model is non-streaming and outputs the speech as a string without diacritical marks. Not recommended for word-for-word transcription and punctuation as accuracy varies based on the characteristics of input audio (unrecognized word, accent, noise, speech type, and context of speech). Since this model was trained on publicly available speech datasets, the performance of this model might degrade for speech which includes technical terms, or vernacular that the model has not been trained on.

How to Use this Model

The model is available for use in the NeMo toolkit [3], and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.

Automatically instantiate the model

import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.EncDecHybridRNNTCTCBPEModel.from_pretrained(model_name="nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0")

Transcribing using Python

First, let's get a sample

wget https://dldata-public.s3.us-east-2.amazonaws.com/2086-149220-0033.wav

Then simply do:

output = asr_model.transcribe(['2086-149220-0033.wav'])
print(output[0].text)

Transcribing many audio files

Using Transducer mode inference:

python [NEMO_GIT_FOLDER]/examples/asr/transcribe_speech.py
 pretrained_name="nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0"
 audio_dir=""

Using CTC mode inference:

python [NEMO_GIT_FOLDER]/examples/asr/transcribe_speech.py
 pretrained_name="nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0"
 audio_dir=""
 decoder_type="ctc"

Training

The [NVIDIA NeMo Toolkit] [3] was used for training the model for two hundred epochs. Model is trained with this example script.

The tokenizer for these model was built using the text transcripts of the train set with this script.

Training, Testing, and Evaluation Datasets

Training Datasets

The model is trained on composite dataset comprising of around 760 hours of Arabic speech:

• Massive Arabic Speech Corpus (MASC) [690h]
  • Data Collection Method: Automated
  • Labeling Method: Automated
• Mozilla Common Voice 17.0 Arabic [65h]
  • Data Collection Method: by Human
  • Labeling Method: by Human
• Google Fleurs Arabic [5h]
  • Data Collection Method: by Human
  • Labeling Method: by Human

Evaluation Datasets

• Massive Arabic Speech Corpus (MASC)
  • Data Collection Method: Automated
  • Labeling Method: Automated
• Mozilla Common Voice 17.0 Arabic
  • Data Collection Method: by Human
  • Labeling Method: by Human
• Google Fleurs Arabic
  • Data Collection Method: by Human
  • Labeling Method: by Human

Test Datasets

• Massive Arabic Speech Corpus (MASC)
  • Data Collection Method: Automated
  • Labeling Method: Aut