Vela-1.0-Omni-Mini

Model overview

Evaluation

Scores are 0–100; higher is better. The comparison uses the same held-out evaluation examples, complete retrieval pools and 128-token text cap for both models. Bold marks improvement over the original large model. These known test pools are reused across releases.

The common protocol uses labeled TRAIN prototypes for text classification and all matching positives for retrieval; it is separate from official MTEB classification. All 14 metrics, Macro-F1, exact counts and uncertainty.

Standard benchmarks and rankings

The primary metric, Mean(TaskType), weights each task type equally. Mean(Task) weights individual tasks equally and is supplementary. Rankings use complete benchmark results from the September 17, 2026 registry snapshots plus both current Vela models; “≤ size” counts models with no more total parameters than this model.

Mini ranks 10/134 on instructed English and 5/50 on audio by Mean(TaskType) among models with no more than its 1.361B total parameters. Its supplementary audio Mean(Task) rank is 3/50. English uses fixed official MTEB task instructions; audio and the scorecard above use the default shared mode. A new comparison with the same text weights and evaluation settings raises Mean(TaskType) from 58.79 to 64.68 (+5.89 pp): 38 tasks improve, three decline, and all seven task types improve. This uses a fresh matched raw control; the historical default result is retained separately. Matched comparison and regressions. Snapshot comparisons include single-modality specialists and reported protocols; the original large has not been evaluated on these complete panels. Full rankings and both aggregate metrics · All task scores and methods.

Audio Mean(TaskType) rises from 52.94 to 54.87 without adding parameters. SpeechCommands zero-shot falls from 40.35 to 38.12, and several speech retrieval and audio classification metrics decline. Every change.

Quality and model size

Each plot combines the full benchmark score, global and size-constrained ranks, and the gap to the best model at no greater total size. Every complete model with a known size is plotted, including models below the observed Pareto frontier. Highlighting Vela does not imply frontier membership. Click either figure for full resolution.

Selected task strengths

These task-level comparisons highlight specific strengths; they do not establish overall benchmark SOTA.

Comparison data and methodology · Figure gallery

Usage

Use PyTorch and a matching torchaudio build, Transformers 4.57.6, Hugging Face Hub, safetensors, NumPy and Pillow. CPU loading was verified with PyTorch/torchaudio 2.8.0:

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("llm-semantic-router/Vela-1.0-Omni-Mini")
sys.path.insert(0, path)
from vela_omni import VelaOmni

model = VelaOmni.from_pretrained(path, device="cpu")
vectors = model.encode_text(["A bicycle beside a building.", "Someone is reading aloud."])
print(vectors.shape)  # (2, 768)

The default mode above keeps text in the shared image/audio space. For text retrieval, opt into an instruction and keep documents unprefixed:

queries = model.encode_text(
    ["How do I reset my password?"],
    instruction="Given a search query, retrieve passages that answer the query.",
    role="query",
)
documents = model.encode_text(["Open Settings and choose Reset password."], role="document")
scores = queries @ documents.T

task="retrieval", role="query" selects a generic preset; custom instruction= and task= are mutually exclusive. The benchmark uses the fixed official task instructions, not a preset sweep. Instructions are intended for text tasks; use the default mode for cross-modal comparisons.

The 32,768-token limit includes the instruction prefix and special tokens. Longer formatted inputs raise ValueError; pass truncate=True on