ARK-ASR

Abstract

ARK-ASR-3B is a 3B-scale audio-capable autoregressive Transformers model for automatic speech recognition.

It combines a Whisper-style audio encoder, an MLP adapter, and a Qwen decoder with custom arkasr remote code.

ARK-ASR currently supports Chinese, English, German, Japanese, French, Korean, Spanish, Polish, Italian, Romanian, Hungarian, Czech, Dutch, Finnish, Croatian, Slovak, Slovene, Estonian, and Lithuanian ASR.

Supported Languages

Chinese, English, German, Japanese, French, Korean, Spanish, Polish, Italian, Romanian, Hungarian, Czech, Dutch, Finnish, Croatian, Slovak, Slovene, Estonian, and Lithuanian.

Model Overview

• Model size: 3B-scale decoder LLM with a dedicated Whisper-style audio encoder and MLP adapter
• Task: automatic speech recognition
• Architecture: audio-capable autoregressive Transformers model with custom arkasr remote code
• Checkpoint format: safetensors
• Sampling rate: 16 kHz
• Recommended inference code: scripts/infer/ark_asr_transformers.py
• vLLM serving: scripts/vllm/ark_asr_vllm

The model should be loaded with trust_remote_code=True. The official inference script handles the processor, tokenizer, audio prompt format, generation cleanup, and ASR token filtering.

Performance

The following results are from the Hugging Face Open ASR Leaderboard. Lower WER is better. ARK-ASR-3B reaches the current state of the art on this English short-form benchmark.

English WER

Chinese CER

Inference

Run ASR inference with Hugging Face Transformers:

import torch
from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer

model_path = "AutoArk-AI/ARK-ASR-3B"
audio_path = "assets/libai.wav"

device = "cuda" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.bfloat16 if device == "cuda" else torch.float32

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    torch_dtype=torch_dtype,
    attn_implementation="sdpa",
).to(device)
model.eval()

def build_bad_words_ids(tokenizer):
    eos_ids = tokenizer.eos_token_id
    keep_ids = {eos_ids} if isinstance(eos_ids, int) else set(eos_ids or [])
    bad_ids = set(tokenizer.all_special_ids) - keep_ids
    bad_ids.update(
        token_id
        for token, token_id in tokenizer.get_added_vocab().items()
        if token.startswith("") and token_id not in keep_ids
    )
    return [[token_id] for token_id in sorted(bad_ids)]

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "path": audio_path},
            {"type": "text", "text": "Please transcribe this audio."},
        ],
    }
]

inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    return_tensors="pt",
    sampling_rate=16000,
    audio_padding="longest",
    text_kwargs={"padding": "longest"},
    audio_max_length=30 * 16000,
)
inputs = inputs.to(device)
if "audios" in inputs:
    inputs["audios"] = inputs["audios"].to(dtype=torch_dtype)

bad_words_ids = build_bad_words_ids(tokenizer)
with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        do_sample=False,
        max_new_tokens=256,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
        bad_words_ids=bad_words_ids,
    )
decoded_outputs = tokenizer.batch_decode(
    outputs[:, inputs.input_ids.shape[1] :],
    skip_special_tokens=True,
)
print(decoded_outputs)

For batch JSONL inference, use the open-source inference code:

git clone https://github.com/AutoArk/open-audio-opd
cd open-audio-opd
pip install -e .

The input JSONL should contain one ASR sample per line:

{"audio":"/path/to/audio.wav","text":"","task":"asr","begin_time":-1,"end_time":-1}

python scripts/infer/ark_asr_transformers.py \
  --input /path/to/input.jsonl \
  --output runs/infer/predictions.jsonl \
  --model_path AutoArk-AI/ARK-ASR-3B \
  --processor_path AutoArk-AI/ARK-ASR-3B \
  --batch_size 40 \
  --dtype bfloat16 \
  --attn_impl sdpa

The output JSONL preserves input metadata and adds:

• pred_text: cleaned prediction text for downstream evaluation
• pred_text_raw: raw decoded generation before cleanup

vLLM Online Serving

ARK-ASR can also be deployed as a vLLM-backed online ASR service with the adapter in [scripts/vllm/ark_asr_vllm](https://github.com/AutoArk/open-audio-op