Phase 7: Speech Synthesis & Applications · ~75 minutes · Python
Streaming Speech-to-Speech — Moshi, Hibiki, and Full-Duplex Dialogue
2024-2026 redefined voice AI. Moshi ships a single model that listens and speaks simultaneously at 200 ms latency. Hibiki does speech-to-speech translation chunk-by-chunk. Both abandon the ASR → LLM → TTS pipeline for a unified full-duplex architecture over Mimi codec tokens. This is the new reference design.
What you will learn
- Implement streaming speech-to-speech — moshi, hibiki, and full-duplex dialogue from scratch
- Understand the math and intuition behind the algorithm
- Use production libraries for the same task
- Ship a reusable artifact
Introduction
- Type: Learn
- Languages: Python
- Prerequisites: Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC and the Semantic-Acoustic Split, Real-Time Audio Processing, The Full Transformer — Encoder + Decoder (Transformers & LLMs from Scratch)
- Time: ~75 minutes
The Problem
Every voice agent built from Lessons 11 + 12 has a fundamental latency floor around 300-500 ms: VAD fires, STT processes, LLM reasons, TTS generates. Each stage has its own minimum latency. You can tune and parallelize, but the pipeline shape caps you.
Moshi (Kyutai, 2024-2026) asks a different question: what if there is no pipeline? What if one model takes audio in and emits audio out directly, continuously, with text as an intermediate "inner monologue" instead of a required stage?
The answer is full-duplex speech-to-speech. Theoretical latency 160 ms (80 ms Mimi frame + 80 ms acoustic delay). Practical latency 200 ms on a single L4 GPU. That's half what a best-in-class pipelined voice agent achieves.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Related lessons
- Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC and the Semantic-Acoustic Split
- Voice Activity Detection & Turn-Taking — Silero, Cobra, and the Flush Trick
- Voice Anti-Spoofing & Audio Watermarking — ASVspoof 5, AudioSeal, WaveVerify
- Audio Evaluation — WER, MOS, UTMOS, MMAU, FAD, and the Open Leaderboards
Browse all courses · View pricing · DeVenture Academy