asr-streaming-conformer-librispeech

This repository provides all the necessary tools to perform automatic speech recognition from an end-to-end system pretrained on LibriSpeech (EN) within SpeechBrain. For a better experience, we encourage you to learn more about SpeechBrain. The performance of the model in full context mode (no streaming) is the following:

With streaming, the results with different chunk sizes on test-clean are the following:

Pipeline description

This ASR system is a Conformer model trained with the RNN-T loss (with an auxiliary CTC loss to stabilize training). The model operates with a unigram tokenizer. Architecture details are described in the training hyperparameters file.

Streaming support makes use of Dynamic Chunk Training. Chunked attention is used for the multi-head attention module, and an implementation of Dynamic Chunk Convolutions was used. The model was trained with support for different chunk sizes (and even full context), and so is suitable for various chunk sizes and offline transcription.

The system is trained with recordings sampled at 16kHz (single channel). The code will automatically normalize your audio (i.e., resampling + mono channel selection) when calling transcribe_file if needed.

Install SpeechBrain

First of all, please install SpeechBrain with the following command:

pip install speechbrain

Please notice that we encourage you to read our tutorials and learn more about SpeechBrain.

Transcribing your own audio files (in English)

from speechbrain.inference.ASR import StreamingASR
from speechbrain.utils.dynamic_chunk_training import DynChunkTrainConfig
asr_model = StreamingASR.from_hparams(
    source="speechbrain/asr-streaming-conformer-librispeech",
    savedir="pretrained_models/asr-streaming-conformer-librispeech"
)
asr_model.transcribe_file(
    "speechbrain/asr-streaming-conformer-librispeech/test-en.wav",
    # select a chunk size of ~960ms with 4 chunks of left context
    DynChunkTrainConfig(24, 4),
    # disable torchaudio streaming to allow fetching from HuggingFace
    # set this to True for your own files or streams to allow for streaming file decoding
    use_torchaudio_streaming=False,
)

The DynChunkTrainConfig values can be adjusted for a tradeoff of latency, computational power and transcription accuracy. Refer to the streaming WER table to pick a value that is suitable for your usecase.

Decoding from a live stream using ffmpeg (BBC Radio 4):

python3 asr.py http://as-hls-ww-live.akamaized.net/pool_904/live/ww/bbc_radio_fourfm/bbc_radio_fourfm.isml/bbc_radio_fourfm-audio%3d96000.norewind.m3u8 --model-source=speechbrain/asr-streaming-conformer-librispeech --device=cpu -v

Decoding from a file:

python3 asr.py some-english-speech.wav --model-source=speechbrain/asr-streaming-conformer-librispeech --device=cpu -v

from argparse import ArgumentParser
import logging

parser = ArgumentParser()
parser.add_argument("audio_path")
parser.add_argument("--model-source", required=True)
parser.add_argument("--device", default="cpu")
parser.add_argument("--ip", default="127.0.0.1")
parser.add_argument("--port", default=9431)
parser.add_argument("--chunk-size", default=24, type=int)
parser.add_argument("--left-context-chunks", default=4, type=int)
parser.add_argument("--num-threads", default=None, type=int)
parser.add_argument("--verbose", "-v", default=False, action="store_true")
args = parser.parse_args()

if args.verbose:
    logging.getLogger().setLevel(logging.INFO)

logging.info("Loading libraries")

from speechbrain.inference.ASR import StreamingASR
from speechbrain.utils.dynamic_chunk_training import DynChunkTrainConfig
import torch

device = args.device

if args.num_threads is not None:
    torch.set_num_threads(args.num_threads)

logging.info(f"Loading model from \"{args.model_source}\" onto device {device}")

asr = StreamingASR.from_hparams(args.model_source, run_opts={"device": device})
config = DynChunkTrainConfig(args.chunk_size, args.left_context_chunks)

logging.info(f"Starting stream from URI \"{args.audio_path}\"")

for text_chunk in asr.transcribe_file_streaming(args.audio_path, config):
    print(text_chunk, flush=True, end="")

We want to optimize some things around the model before we create a proper HuggingFace space demonstrating live streaming on CPU.

In the mean time, this is a simple hacky demo of live ASR in the browser using Gradio's live microphone streaming feature.

If you run this, please note:

• Modern browsers refuse to stream microphone input over an untrusted connection (plain HTTP), unless it is localhost. If you are running this on a remote server, you could use SSH port forwarding to expose the remote's port on your machine.
• Streaming using Gradio on Firefox seems to cause some issues. Chromium-based browsers seem to behave better.

Run using:

python3 gradio-asr.py --model-source speechbrain/asr-streaming-conformer-librispeech --ip=localhost --device=cpu

from argparse import ArgumentParser
from dataclasses im