MERT-FullSong

🚀 Quick start

Install the matching PyTorch packages:

python -m pip install torch==2.6.0 torchaudio==2.6.0 transformers==4.53.2 huggingface-hub safetensors soundfile

import soundfile as sf
import torch
import torchaudio.functional as AF
from transformers import AutoFeatureExtractor, AutoModel

repo = "m-a-p/MERT-v2-FullSong"
device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoFeatureExtractor.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval().to(device)

audio, sr = sf.read("music.wav", dtype="float32", always_2d=True)
audio = audio[:30 * sr].mean(axis=1)  # First 30 seconds, mixed to mono.
waveform = AF.resample(torch.from_numpy(audio), sr, processor.sampling_rate)
inputs = processor(
    waveform.numpy(), sampling_rate=processor.sampling_rate, return_tensors="pt",
).to(device)

with torch.inference_mode():
    output = model(**inputs, output_hidden_states=True)

frames = output.last_hidden_state         # [batch, frames, 1024], 25 Hz
layers = output.hidden_states             # 24 tensors, one per block
mask = output.feature_attention_mask[..., None]
embedding = (frames * mask).sum(1) / mask.sum(1).clamp_min(1)  # [batch, 1024]

hidden_states[0] is block 1; hidden_states[23] is block 24. Remove the audio slice to process a complete song.

📊 MARBLE

Reported frozen-encoder results on MARBLE. All scores are multiplied by 100; higher is better. Bold marks the best displayed value in each column, including ties.

Baseline scores and parameter counts are reproduced from Tables III and V of PupuJEPA; the baselines were not rerun for this release. Evaluation protocols may differ across sources.

General music understanding. MTT = MagnaTagATune; key = GiantSteps refined key accuracy; genre and beat = GTZAN; valence and arousal = EmoMusic. ROC = ROC-AUC; AP = average precision.

MTG-Jamendo tagging. Mood denotes mood/theme tags.

Supplemental: Chords1217 frame accuracy is 78.48 for MERT-v2-30s and 77.79 for MERT-v2-FullSong.

HF reproduction verified: Both models reproduce all ten MARBLE tasks with the fixed evaluation settings. Scores and best settings.

🎛️ Layer guide

Recommended probe settings with the encoder frozen. L1 = hidden_states[0]; All-layer MLP uses all 24 layers.

📚 Citation

Technical report coming soon. For now, please cite [MERT (ICLR 2024)](https://proceedings.iclr.cc/paper_files/paper/2024/hash/33dffa2