Phase 7: Speech Synthesis & Applications · ~75 minutes · Python
Streaming Speech-to-Speech — Moshi, Hibiki, and Full-Duplex Dialogue
2024-2026 redefined voice AI. Moshi ships a single model that listens and speaks simultaneously at 200 ms latency. Hibiki does speech-to-speech translation chunk-by-chunk. Both abandon the ASR → LLM → TTS pipeline for a unified full-duplex architecture over Mimi codec tokens. This is the new reference design.
Hiring signal: Understanding of streaming speech-to-speech — moshi, hibiki, and full-duplex dialogue internals
What you will learn
- Implement streaming speech-to-speech — moshi, hibiki, and full-duplex dialogue from scratch
- Understand the math and intuition behind the algorithm
- Use production libraries for the same task
- Ship a reusable artifact
Introduction
Type: Learn Languages: Python Prerequisites: Phase 6 · 13 (Neural Audio Codecs), Phase 6 · 11 (Real-Time Audio), Phase 7 · 05 (Full Transformer) Time: ~75 minutes
The Problem
Every voice agent built from Lessons 11 + 12 has a fundamental latency floor around 300-500 ms: VAD fires, STT processes, LLM reasons, TTS generates. Each stage has its own minimum latency. You can tune and parallelize, but the pipeline shape caps you.
Moshi (Kyutai, 2024-2026) asks a different question: what if there is no pipeline? What if one model takes audio in and emits audio out directly, continuously, with text as an intermediate "inner monologue" instead of a required stage?
The answer is full-duplex speech-to-speech. Theoretical latency 160 ms (80 ms Mimi frame + 80 ms acoustic delay). Practical latency 200 ms on a single L4 GPU. That's half what a best-in-class pipelined voice agent achieves.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Use It, Pitfalls, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Related lessons
- Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC and the Semantic-Acoustic Split
- Voice Activity Detection & Turn-Taking — Silero, Cobra, and the Flush Trick
- Voice Anti-Spoofing & Audio Watermarking — ASVspoof 5, AudioSeal, WaveVerify
- Audio Evaluation — WER, MOS, UTMOS, MMAU, FAD, and the Open Leaderboards
Browse all courses · View pricing · DeVenture Academy