moshika-rag-pytorch

About

What moshika-rag-pytorch is

Model Card for MoshiRAG MoshiRAG is a compact full-duplex speech language model augmented with asynchronous knowledge retrieval to improve factuality without sacrificing real-time interactivity. Built on top of Moshi, MoshiRAG predicts when a query needs external knowledge, retrieves references within the natural response delay, and grounds answers in stronger knowledge sources. Model Details Pytorch version with bf16 precision. Model Description MoshiRAG uses a modular front-end/back-end design: - Front-end is a full-duplex speech model based on Moshi that handles real-time conversation. - Back end is an asynchronous retrieval system running in parallel to fetch factual information when needed. The front end keeps listening and speaking continuously. When the model predicts a retrieval trigger token, conversation context is sent to the retrieval back end while the dialogue continues. During this period, the model can produce lightweight pre-RAG content (for example, short acknowledgments or coarse responses) so the interaction stays natural. The back end is text-in/text-out and can be implemented with different retrieval methods (LLM-based retrieval or search-based retrieval, etc). The retrieval back end takes conversation context (derived by combining the text predicted by Moshi inner monologue and the user transcription predicted by a streaming ASR component) as inputs, and then returns the reference text. Once the retrieval is completed, the reference text is encoded and injected back into Moshi as a stream, allowing later response segments to be grounded in external knowledge without interrupting the ongoing conversation. This repository contains the front-end model only. Please refer to the Github repository for more details about various choices of retrieval back ends. - Developed by: Kyutai - Model type: Multimodal speech-text foundation model with additional text conditioning - Language(s) (NLP): English - License: CC-BY 4.0 - Dependency: MoshiRAG uses frozen components from ARC-Encoder. An additional Streaming ASR model is required to transcribe the user speech so as to provide conversation context to the retrieval back end. Model Sources - Repository:

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How audio & music work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys moshika-rag-pytorch for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (moshika-rag-pytorch below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/audio/transcriptions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -F model="moshika-rag-pytorch" -F file=@audio.mp3

Details

Languages, data & research

Papers

Licence

Open weights

Sources

Weights & code

Explore

More audio & music