BotNetwork

Guide 17

Voice bots: speech-to-text, text-to-speech, telephony and latency budgets

In voice, every pause is noticed. The architecture is simple; making it feel natural is about streaming and timing.

Guide2 min readUpdated

The pipeline

Most voice bots chain three stages:

  1. Speech-to-text (STT) turns audio into text, ideally streaming partial results.
  2. Dialogue logic: rules, a state machine or an LLM decides the reply. See AI chatbots and agents.
  3. Text-to-speech (TTS) turns the reply into audio, ideally streaming the first audio chunk before the whole sentence is generated.

Some providers offer speech-to-speech models that merge the stages. They cut latency but give you less control over what is said, which matters for compliance.

Latency

People expect a reply roughly as fast as another person would answer, often within about a second. Measure each stage separately: end-of-speech detection, STT finalization, first token from the model, first audio byte from TTS and network time. Common techniques:

  • Stream everything and start TTS on the first complete sentence.
  • Keep prompts short and use smaller models for simple turns.
  • Play a short acknowledgment if a tool call will take long.
  • Host services close to each other and to the telephony provider.

Turn-taking

  • Voice activity detection (VAD) decides when the person has finished. Too eager and you interrupt; too slow and the bot feels laggy.
  • Barge-in lets the caller interrupt: stop playback immediately when speech is detected and discard the rest of the reply.
  • Confirm critical values back ("I heard 4 5 1 2, is that right?") because STT errors on numbers and names are common.

Channels

  • Phone: telephony providers connect calls to your service via SIP or a media WebSocket that streams audio frames (often 8 kHz mu-law). Handle DTMF key presses as a fallback.
  • Web and apps: WebRTC gives low-latency audio with echo cancellation built in.
  • Chat apps: Telegram, WhatsApp and Discord deliver voice messages as audio files (commonly Ogg/Opus). Download, transcribe and reply in text or audio. Discord voice channels need a gateway voice connection, which libraries such as discord.js voice support.
  • Tell callers they are talking to an automated system, and offer a human.
  • Recording and transcribing calls is regulated differently by jurisdiction; some require consent from every party. Announce recording and get legal advice for your markets. See legal, terms and privacy.
  • Do not clone a real person's voice without explicit permission.
  • Keep audio only as long as needed; transcripts are personal data too.

Testing

Build a set of recorded utterances with accents, background noise and numbers, and track word error rate and end-to-end latency over time. See testing bots.

Check your understanding

1. What is barge-in?
2. Which technique lowers perceived latency most?