How Voice AI Learned to Know When You're Done Talking

Five decades of turn-taking research, mapped: from conversation analysis to predictive models to full-duplex speech. A living timeline; if something is wrong or missing, tell me and I will fix it.

The 200 millisecond problem

In human conversation, the gap between speakers averages about 200 milliseconds, across languages and cultures. That is faster than human reaction time, which means people do not wait to hear you stop talking. They predict when you will stop and plan their reply while you are still speaking.

Production systems could not reliably do this until roughly two years ago. This page traces how the field learned, through the research literature: the conversation scientists who described the mechanism fifty years ago, the dialogue researchers who turned it into models, and the full-duplex systems that are now changing the question entirely. It is the map I wanted when I started working on this problem.

Five answers to one question

Every approach to turn-taking is an answer to "what signal tells you the turn is over?" The field's history is a walk down this table.

Era 1: Humans figure it out first (1970-2015)

Everything voice AI rediscovered about turn-taking was published between 1970 and 2015, most of it before deep learning existed. The core findings: turn-ends are projected in advance, not detected from silence; the cues are syntactic and prosodic; and listeners plan their response during the incoming turn. This era is the answer key the field graded itself against later.

1. 1970HUMANSYngve

   On getting a word in edgewise

   Names the backchannel: "mm-hmm" is not a turn, it is a signal to keep going. Machines still confuse the two today.

2. 1972HUMANSDuncan · JPSP

   Some signals and rules for taking speaking turns in conversations

   Identifies discrete turn-yielding cues: pitch movement, drawl on the final syllable, syntactic completion, gesture. The first inventory of the signal machines would later learn from data.

3. 1974HUMANSSacks, Schegloff and Jefferson · Language

   A simplest systematics for the organization of turn-taking for conversation

   The founding paper of conversation analysis. Turns are built from projectable units; listeners anticipate completion points and aim for them. One of the most cited papers in the social sciences, and the conceptual spec for every predictive turn-taking model since.

4. 2006HUMANSDe Ruiter, Mitterer and Enfield · Language

   Projecting the end of a speaker's turn

   Experiments splicing prosody and words apart show listeners project turn-ends primarily from lexico-syntactic content. Foreshadows why text-based end-of-turn models work as well as they do.

5. 2009HUMANSStivers et al. · PNAS

   Universals and cultural variation in turn-taking in conversation

   Ten languages, five continents: the ~200ms gap is a human universal, with only small cultural variation. This is the latency target, published before anyone could hit it.

6. 2015HUMANSBögels and Torreira · Journal of Phonetics

   Listeners use intonational phrase boundaries to project turn ends

   Prosody matters exactly at the ambiguous points: intonation tells listeners whether a syntactically complete utterance is actually done. The case for audio-native detection, made a decade early.

7. 2015HUMANSLevinson and Torreira · Frontiers in Psychology

   Timing in turn-taking and its implications for processing models of language

   Producing even a short utterance takes around 600ms of planning, so comprehension and production must overlap. Humans are full-duplex. This paper is the endpoint of the human thread and the quiet thesis of this whole page.

You just read what humans do. Here is what machines did instead, on one utterance:

Era 2: Machines use a stopwatch (1997-2016)

While the human literature was converging on prediction, deployed systems used voice activity detection plus a silence timer: the turn is over after N milliseconds of quiet. Everyone knew it was wrong; the interesting work in this era is the attempts to make the timer smarter rather than replace it.

1. 1997-2011MACHINESTelephony and WebRTC

   Energy-based and statistical VAD

   WebRTC ships a lightweight GMM-based voice activity detector that becomes the telephony default. It detects sound, not turns. The silence threshold tuned on top of it is unwinnable: short values interrupt, long values lag.

2. 2000MACHINESWard and Tsukahara · Journal of Pragmatics

   Prosodic features which cue back-channel responses

   Low pitch regions predict backchannel opportunities in English and Japanese. Early proof that prosody carries turn-taking information a VAD throws away.

3. 2008MACHINESRaux and Eskenazi · SIGDIAL

   Optimizing endpointing thresholds using dialogue features

   In the Let's Go bus information system, endpointing thresholds adapt per dialogue state and utterance features. The smartest version of the stopwatch, and an admission that one number cannot work.

4. 2011MACHINESGravano and Hirschberg · Computer Speech and Language

   Turn-taking cues in task-oriented dialogue

   Seven measurable turn-yielding cues; the more cues present, the more likely a turn shift. Duncan 1972, now with feature vectors.

5. 2011MACHINESSchlangen and Skantze · Dialogue and Discourse

   A general, abstract model of incremental dialogue processing

   Systems should process and react while the user speaks, not after. The architectural argument that makes prediction usable.

Era 3: Prediction replaces detection (2017-2023)

The deep learning era finally implements what the 1974 paper described. The reframe: do not classify silences, continuously predict upcoming speech activity for both speakers. This is the line of work I know best; the group that built VAP at KTH is the group I am joining for my master's thesis.

1. 2017MACHINESSkantze · SIGDIAL

   Towards a general, continuous model of turn-taking using LSTM recurrent neural networks

   An LSTM trained on raw dialogue predicts future voice activity continuously, learning turn-taking end to end instead of from hand-built cues. The pivot paper for the machine thread.

2. 2018MACHINESRoddy, Skantze and Harte · ICMI

   Multimodal continuous turn-taking prediction using multiscale RNNs

   Acoustic and linguistic features at multiple timescales; shows which cues matter at which horizons for continuous prediction.

3. 2020PRODUCTIONSilero

   Silero VAD

   Not a turn-taking model, but the small neural VAD that becomes the substrate nearly every production pipeline still runs on.

4. 2021MACHINESSkantze · Computer Speech and Language

   Turn-taking in conversational systems and human-robot interaction: a review

   The review that organizes the whole field. If you read one survey, read this one.

5. 2022MACHINESEkstedt and Skantze · Interspeech

   Voice Activity Projection: self-supervised learning of turn-taking events

   Rather than deciding whether the current silence means done, VAP listens to both channels of a conversation and continuously predicts the probability of each speaker's voice activity over the next two seconds. Turn shifts, backchannels and overlap handling all read out of one representation. The modern reference model for predictive turn-taking.

6. 2022MACHINESEkstedt and Skantze · SIGDIAL

   How much does prosody help turn-taking?

   Probing shows VAP genuinely uses intonation, validating Bögels and Torreira inside a neural model.

Era 4: The literature ships (2024-2026)

Two things happen at once. In research, VAP goes real-time, multilingual, multi-party and multimodal. In industry, every serious voice platform replaces the silence timer with a learned end-of-turn model. The vendors mostly do not cite the literature, but the three camps they split into, transcript-based, audio-native, and integrated into the recognizer, map exactly onto the cue debates the human thread settled decades ago: De Ruiter's syntax, Bögels' prosody, or both.

A note on sourcing: cards tagged PRODUCTION describe shipped systems and engineering-blog claims, not peer-reviewed results. The colored tags keep the two visibly separate.

1. 2024MACHINESInoue, Jiang, Ekstedt, Kawahara and Skantze · IWSDS

   Real-time and continuous turn-taking prediction using Voice Activity Projection

   VAP runs live on CPU, closing the gap between the research instrument and a deployable component.

2. 2024MACHINESInoue et al. · LREC-COLING

   Multilingual turn-taking prediction using Voice Activity Projection

   English, Mandarin and Japanese: turn-taking prediction transfers across languages with language-specific fine-tuning.

3. 2024-2025PRODUCTIONLiveKit, Pipecat, Deepgram, AssemblyAI, OpenAI

   Semantic endpointing ships everywhere

   LiveKit's open-weights transcript-based turn detector; Pipecat's Smart Turn, open source and audio-native, CPU real-time by v3; Deepgram Flux folds eager end-of-turn into the recognizer so LLM calls start speculatively; AssemblyAI and OpenAI's Realtime API ship semantic endpointing modes. The silence timer quietly stops being the default.

4. 2025MACHINESElmers, Inoue, Lala and Kawahara · Interspeech

   Triadic multi-party Voice Activity Projection

   Prediction extends to three-party conversation, where next-speaker selection becomes part of the problem.

5. 2025-2026MACHINESMultiple groups

   VAP goes multimodal and in-the-wild

   Visual cues stabilize prediction in noise; pretrained voice-activity encoders and multiparty in-the-wild variants push past clean two-party audio; field experiments put noise-robust VAP on real dialogue robots.

Era 5: Full duplex dissolves the question (2022-)

The radical answer: stop deciding when the user is done and build models that listen while speaking, the way Levinson and Torreira showed humans do. "Has the turn ended?" becomes a continuous policy inside the model: stay silent, backchannel, or take the floor, decided frame by frame. The open problem shifts from detection to evaluation, and the architecture question splits the field into engineered versus learned synchronization.

1. 2022DUPLEXNguyen et al. · Meta AI

   Generative spoken dialogue language modeling (dGSLM)

   Dialogue generated on two parallel audio channels: overlaps, laughter and backchannels emerge from data. The proof of concept for duplex generation.

2. 2024DUPLEXMa et al.

   Language model can listen while speaking

   Gives the capability its name and a minimal architecture for it.

3. 2024DUPLEXVeluri et al. · EMNLP

   Beyond turn-based interfaces: synchronous LLMs as full-duplex dialogue agents

   A text LLM adapted into a synchronous full-duplex agent. Duplex behavior without abandoning the LLM stack.

4. 2024DUPLEXDéfossez et al. · Kyutai

   Moshi: a speech-text foundation model for real-time dialogue

   The landmark: parallel streams for both speakers, theoretical response latency under 200ms. The human number, finally on the board.

5. 2025DUPLEXMultiple groups

   The benchmarks arrive

   Talking Turns probes audio foundation models on turn-taking dynamics; Full-Duplex-Bench and its v2 evaluate pause handling, backchanneling and interruption; FLEXI benchmarks human-LLM duplex interaction.

6. 2025DUPLEXSurvey

   From turn-taking to synchronous dialogue

   The field gets its taxonomy: engineered synchronization (modular systems with explicit turn logic) versus learned synchronization (end-to-end duplex models). The central design question, named.

7. 2026PRODUCTIONOpenAI · July 2026

   GPT-Live-1: full duplex reaches the consumer default

   ChatGPT's voice now backchannels while you talk. OpenAI has not published the architecture; public reporting describes a fast conversational model with asynchronous handoff to a stronger reasoning model. If that description holds, it is engineered and learned synchronization in one product. Either way, turn-taking is now a product spec, not a research curiosity.

What the map says

• The human literature was the spec all along. Every machine-thread milestone implements a finding published decades earlier: projection (1974) became continuous prediction (2017), cue inventories (1972, 2011) became feature learning, overlap of comprehension and production (2015) became full duplex (2024).
• Prosody keeps getting rediscovered. De Ruiter said syntax suffices, Bögels showed prosody decides the hard cases, and the current transcript-versus-audio split in production detectors is the same argument with venture funding.
• Evaluation is the open frontier. End-of-turn precision does not measure conversational feel; duplex benchmarks are one year old and unsettled. Whoever defines the metrics defines the field.
• The engineered-versus-learned synchronization question is still open. Public reporting on GPT-Live-1 suggests the answer may be layered. Whether a fully modular cascade, with components you can inspect and swap, can reach duplex behavior at near-zero latency is the live question.

Why I care

I build real-time voice systems in production as co-founder of Flowent, a live AI language tutor, and from 2027 I will work on exactly that last question, full-duplex cascaded speech-to-speech, for my master's thesis at KTH Speech, Music and Hearing with Prof. Gabriel Skantze, whose group created VAP. This page is the map I navigate by, and it will keep changing as the field does.