Skip to main content
Knowing exactly when the caller has finished speaking is the hardest part of a real-time voice agent. Answer too early - you talk over them. Too late - the agent sounds sluggish. The platform combines three complementary layers to get this right.

​
1. VAD - voice activity detection (Silero)

The transport layer runs every audio chunk through a local Silero VAD model (ONNX), which answers a simple question for each frame:
“Is this speech, or silence?”
Configured in the agent’s vad_configuration: VAD is responsible for interruption detection (the caller speaking while the bot speaks) and is the base signal for the turn analyzer.
A bare “yes” or “no” lasts about 0.2–0.4 s. With start_seconds at 0.4 (the former default, still stored on agents created earlier) VAD often misses such a reply and the agent reacts to it late. If your callers answer in one word, set 0.2. A higher value only makes sense on noisy lines, where short clicks interrupt the agent.
On the Accurate transcription tier (Soniox) it is VAD that closes an utterance. When VAD misses it, the transcript is closed on its own after 0.5 s without new words, so a short reply does not wait for the caller’s next sentence.

​
2. Deepgram streaming parameters

Deepgram (STT over WebSocket) receives audio in parallel with VAD and produces two kinds of transcripts:
  • Interim - partial, updated many times per second.
  • Final (is_final=true) - a committed fragment that won’t change.
Two parameters decide when Deepgram “closes off”:

​
endpointing (ms)

The silence threshold that marks a phrase boundary - after what pause Deepgram commits the current transcript (is_final=true). Lower values = faster commits, but a single thought may split into several segments.
  • Range: 10–1000 ms
  • Default: 100 ms (fast)
  • Higher values (300–500 ms) for callers who pause mid-sentence.

​
utterance_end_ms (ms)

The silence threshold that marks the end of the whole utterance - after what silence Deepgram emits the UtteranceEnd event, which triggers the LLM.
  • Minimum: 1000 ms - a hard Deepgram requirement. Values below it cause the WebSocket connection to be rejected with HTTP 400, and STT never connects. The backend automatically bumps older configurations up to 1000.
  • Range: 1000–5000 ms
  • Default: 1000 ms

​
3. Smart Turn V3 - ML end-of-turn detection

Silence timers alone aren’t enough - people pause for many reasons (thinking, breathing, reading out a number), and 1000 ms is an eternity in conversation. Smart Turn V3 is a local ONNX model that analyzes the prosody of the utterance - intonation, rhythm, the final pitch contour - and returns the probability that the turn is finished. When enabled on the agent (smart_turn_enabled: true):
  • after every VAD silence boundary the audio segment goes through the model,
  • the model returns {prediction: 0|1, probability: 0..1},
  • prediction == 1 → the LLM starts immediately, often hundreds of ms before the utterance_end_ms timeout,
  • prediction == 0 → the pipeline keeps listening; utterance_end_ms acts as a safety net.

​
Does utterance_end_ms still matter with Smart Turn enabled?

Yes. It’s a “belt and suspenders” setup: Lowering utterance_end_ms below 1000 ms would speed up the fallback path, but Deepgram doesn’t allow it. If you want lower latency - enable Smart Turn.

​
When the caller talks over the agent

The caller can interrupt the agent at any moment - the agent stops and listens. So that an “mhm” or “uh-huh” dropped while the agent is talking does not cut its sentence short, an interruption during the agent’s speech counts only from two words of the caller. “No, no” or “wait, wait” interrupts at once; a single nod is ignored and the agent carries on. While the agent is silent, every word of the caller counts as before. The threshold is the Interruption sensitivity slider in the agent’s settings: at maximum any sound interrupts (an “mhm” too, as before), in the middle two words, at minimum four. Lower sensitivity suits noisy lines where the VAD catches fragments of words; higher, callers who interrupt with a single word (“stop”, “hello”). The API also takes an explicit workflow_configurations.interruption_min_words (0 = any sound). A short answer under the threshold is not lost. When the caller drops a “Yes” over the end of “shall I book you for Tuesday?”, the agent does not cut its sentence, but it keeps that “Yes”, and if it finishes speaking within 1.5 s it treats it as the caller’s ordinary reply. A nod in the middle of a longer sentence (the agent goes on for more than 1.5 s) is still ignored. If the caller goes on right after the short word and interrupts the agent, both parts reach the model as one utterance. Change the window with workflow_configurations.short_reply_hold_seconds (0 turns it off, maximum 5); with the slider at maximum there is no threshold, so nothing needs keeping.

​
How long the caller really waits

In a call transcript, each agent reply carries a time with a stopwatch icon: from the moment the caller went quiet to the agent’s first audio. It is made of the wait for the end of speech (stop_seconds), transcription, the LLM and the start of speech synthesis. On the phone the network adds its own delay, usually 0.2–0.4 s, which the platform does not measure. Calls from before this measurement show a brain icon and the model’s thinking time alone, which is only part of that pause.
  • Natural conversation (recommended): Smart Turn ON, utterance_end_ms = 1000, endpointing = 100.
  • Dictating numbers / long pauses: Smart Turn ON, utterance_end_ms = 1500–2000, endpointing = 300.
  • Minimum latency without Smart Turn: utterance_end_ms = 1000, endpointing = 100 - occasional talk-overs possible.