bagpiper

Bagpiper-Base is the pretrained 8B checkpoint from Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions. It learns a bidirectional mapping between physical audio and detailed natural-language descriptions spanning speech, music, environmental sound, and mixtures of them.

This is the Base checkpoint, not the instruction-tuned assistant. Use espnet/bagpiper-sft for the caption-then-process model trained to answer open-ended audio requests. Bagpiper-Base is intended for audio-to-rich-caption probing, rich-caption-conditioned audio generation, and downstream initialization.

Repository contents

base.pt is a PyTorch mapping with model state under the top-level module key. It contains 1,382 tensors: 1,381 BF16 tensors from training and one deterministic FP32 vocab_weight buffer required by the current strict ESPnet loader. It contains no optimizer, scheduler, or exact-resume state. This is not a Transformers from_pretrained directory, and no vLLM compatibility is claimed.

The YAML reconstructs the architecture and vocabulary of the released checkpoint. The original stage-2 launch referenced this filename but did not commit it, so its trainer fields should not be treated as an archival record of every training setting.

Download and integrity

hf download espnet/bagpiper \
  --revision d7b91ab261116973fb65cf44ff9590f6f242323b \
  --local-dir bagpiper-base

echo "c9917c1237bd44ea0ea8b41746e5ec5aecae92273538c9b6e4040d26f69a6521  bagpiper-base/base.pt" \
  | sha256sum -c -

Architecture

The audio encoder converts input waveforms into continuous representations. X-Codec provides discrete output tokens that can be decoded back into audio. Expanded text and audio embeddings allow the Qwen3 backbone to model both directions through one shared sequence interface.

ESPnet runtime

The native consumer is espnet2/speechlm/bin/inference.py. Reconstruct the model with the supplied training YAML, pair it with the task-appropriate inference YAML from the Bagpiper recipe, and pass base.pt through --model-checkpoint.

The released state dict was loaded through ESPnet's strict inference path on an NVIDIA H100 using ESPnet PR #6503 at commit 1992821bfb6d68c6636d1b75340b8b2b591bc015 and Transformers 5.5.4. A complete clean-room installation and end-to-end quickstart have not yet been validated; the Hub Inference API is therefore disabled. The paper estimates roughly 24 GB of GPU memory for BF16 inference, with longer inputs, decoding, or additional workers requiring more.

Pretraining

Bagpiper-Base is the result of the first two stages of the paper's curriculum: connector and embedding warmup followed by full pretraining. The audio encoder and codec remain frozen during full pretraining while the backbone, connectors, and expanded vocabulary are optimized. The paper reports training on 80 NVIDIA GH200 GPUs.

The 600B figure in the paper is a token budget, not a row count: 300B text-to-audio, 150B audio-to-text, and 150B text-only tokens. The audio pipeline began from roughly 422M raw audio-caption pairs, capped clips at 30 seconds, and used machine-generated rich captions plus quality, alignment, and deduplication filters.

The associated public Bagpiper_PreTrain_Data snapshot is a large artifact from this pipeline, not a claim of a complete one-to-one dump of every raw pair or text-only source used in training.

Paper-reported Base probes

These results evaluate the pretrained audio-rich-caption mapping. They were not rerun as part of this Hub card update and must not be interpreted as end-to-end task-solving scores.

For the understanding probes, an external text LLM answers using only Bagpiper-Base's generated caption; the model does not solve the final task itself. The speech-generation WER uses only the roughly 60% of test-clean examples whose conditioning rich captions preserve the full transcription. Generation baselines receive transcripts or short captions, whereas Bagpiper-Base receives full rich captions. See the paper for the complete protocol, comparisons, cycle-consistency results, and text-capability probes.

Intended use

• Research on bidirectional audio-rich-caption representations.
• Rich-caption generation for analysis or as input to a separately evaluated downstream system.
• Rich-caption-conditioned generation of speech, music, sound, and mixtures.
• Initialization for downstream ESPnet SpeechLM fine-tuning.

For general instruction following and open-ended audio question answering or gene