vui

It ships inside Vui, a real-time voice assistant: speak into your mic, the model transcribes, runs a local LLM, and streams a TTS reply back — all from a single Python server.

Features

• Vui Nano (219M active, 305M total) — a small, context-aware TTS model: Llama-style decoder + RQ-Transformer head over the Qwen3-TTS-12Hz codec, Apache 2.0
• Conversation-conditioned generation — replies are decoded from a KV cache holding the whole dialogue, including the audio of your turn (~6-minute context)
• CPU inference, zero dependencies — a pure-C engine: one binary plus one weight file, no Python, PyTorch, or ONNX at runtime; supports voice cloning and streaming playback
• Real-time voice loop — WebRTC + WebSocket pipeline (ASR → LLM → TTS) with a browser UI, VAD-driven turn taking, speculative LLM prefill while you're still speaking, sentence-level TTS chunking with backpressure
• Barge-in — start talking mid-reply, the model cancels and listens
• Streaming TTS — ~9× realtime on a 4090, bf16 inference, CUDA graphs
• OpenAI Realtime API compatible — drop-in ws://…/v1/realtime for clients written against OpenAI's spec (docs/realtime-api.md)
• One-shot voice-note REST endpoint — POST /v1/voice-note runs the whole ASR → LLM → TTS pipeline in a single HTTP call (audio in, JSON out)
• Standalone TTS demo — demo.py Gradio playground for the model on its own
• Voice cloning — upload an audio sample to clone any speaker; 4 fine-tuned presets shipped (maeve, abraham, rhian, harry)
• SQ / WPS conditioning — bias generation on six speech-quality channels and words-per-second
• Hot-swap models — pick Ollama LLM and ASR backend live from the UI
• Pluggable ASR — faster-whisper (GPU) or Moonshine (CPU streaming, ONNX)
• Pluggable LLM backends — Ollama, vLLM, any OpenAI-compatible endpoint
• Memories — assistant remembers facts about you across sessions
• Thoughts stream — parallel LLM routes voice intent to ~10 tools (memory ops, task control, delegation) without a wake-word grammar; pluggable for your own local tools
• Optional Claude task server — sidecar agent that handles slow/agentic work (Gmail, Calendar, Drive, Slack, web search) via your existing Claude Code MCPs
• Apple Silicon support — MLX backend (WIP)
• Mobile-ready — documented cloudflared and Tailscale paths for phone access with mic over HTTPS
• Docker compose — one file brings up the full stack
• OpenClaw integration — point OpenClaw's openai realtime provider at Vui for a fully-local voice front-end

Install (one-liner)

curl -fsSL https://install.fluxions.ai | bash

Clones into ~/vui, auto-detects Docker vs. native, installs deps (uv, Ollama, ffmpeg, Claude Code CLI), pulls the Ollama LLM, and launches the stack on .

Full Docker compose / native install, mobile setup, configuration, ASR options, and the Claude task server — all in the GitHub README.

TTS demo on its own

git clone https://github.com/fluxions-ai/vui
cd vui
uv sync
python demo.py                                          # Gradio UI — upload your own voice prompt
python demo.py --render --prompt prompts/abraham.wav    # CLI render with a preset voice

The Vui checkpoint and Qwen codec download automatically from this repo on first run.

Preset voices

More personalities coming soon! Got a voice or character you'd like to hear? Open an issue or let us know on Discord.

Python API

from vui.engine import Engine, GenConfig

engine = Engine()  # vui-nano-1.1 by default; Engine("vui-190k") / Engine("vui-nano") for the others
with engine.new_row() as row:
    codes, audio = row.render(   # render() returns (codes (T,Q), audio (1,1,S))
        "So [breath] the thing about this is, it's not what you'd expect, right?",
        GenConfig(temperature=0.7),
    )

Clone a voice

Cloning is a prefill: hand the model Segment(text, codes) — a reference transcript plus its encoded audio — and everything rendered afterwards follows that speaker.

import torch
from julius.resample import resample_frac
from torchcodec.decoders import AudioDecoder
from torchcodec.encoders import AudioEncoder

from vui.engine import Engine, Ge