Real-Time Voice AI Pipelines for Phone Calls: Streaming STT, LLM, TTS & Barge-In

Building a voice AI prototype in a browser tab takes an afternoon. Making that same voice agent hold a natural conversation over a live telephone line is a distributed systems engineering problem.

The fix is a pipelined architecture where every downstream stage starts working on partial output before the upstream stage finishes. Concretely, that means wiring together five pieces end to end:

Bridging narrowband PSTN/SIP telephony streams to full-duplex WebSockets.
Combining local audio Voice Activity Detection (VAD) with word-timing endpointing so noisy phone lines do not break turn detection.
Streaming LLM tokens into TTS at safe syntactic clause boundaries without clipping prosody or choking on abbreviations.
Executing full-stack barge-in by flushing telephony gateway playback buffers and truncating conversation history to what the caller actually heard.
Decoupling synchronous webhook and CRM tool calls from the audio loop.

In human conversation, turn-taking is remarkably fast. A cross-linguistic study by Stivers et al. (2009) in the Proceedings of the National Academy of Sciences found that across ten languages, the cross-language mean response offset was about 208 milliseconds, while the overall modal (most common) offset was 0 milliseconds, meaning the single most frequent pattern was the next speaker starting at almost the same instant the current speaker finished, not after a gap. Because preparing a single spoken word takes roughly 600 milliseconds before articulation begins (an estimate from picture-naming studies, reviewed by Levinson and Torreira (2015) in Frontiers in Psychology), human listeners do not wait for silence to start thinking; they overlap comprehension with response formulation while the other person is still speaking.

In telecommunications network planning, ITU-T Recommendation G.114 recommends that one-way transmission delay for general network planning should not exceed 400 milliseconds, while cautioning that highly interactive tasks can be perceptibly affected by much lower delays than that ceiling.

When you place an AI pipeline in the middle of a phone call, every millisecond of silence after the caller stops speaking adds to that turn-taking gap. If your backend waits for Speech-to-Text (STT) to finalize a full paragraph, then waits for a Large Language Model (LLM) to generate a complete answer, and finally waits for Text-to-Speech (TTS) to render a full audio file before streaming bytes back to the carrier gateway, the caller hears dead air and assumes the call dropped.

At Rymi, we build voice AI infrastructure that exposes one API for STT, LLM, TTS, and telephony, with voices for English and Hindi, including Hinglish. While our earlier guide on production voice AI agents covered provider trade-offs and failure fallbacks, this deep dive focuses on the protocol and orchestration mechanics behind the five pieces above.

1. The Telephony-to-Model Pipeline: What Happens on a Phone Call

A browser-based voice demo captures wideband microphone audio (typically 16 kHz or 48 kHz) over WebRTC. A standard public switched telephone network (PSTN) call operates under much stricter physical and protocol constraints:

Call Signaling (IETF RFC 3261): Session Initiation Protocol (SIP) negotiates the call session, codecs, and media endpoints between the carrier and your Session Border Controller (SBC) or telephony provider (such as Twilio, Plivo, Vonage, or Telnyx).
Media Transport (IETF RFC 3550): Once the call connects, audio frames travel over the Real-time Transport Protocol (RTP), carrying sequence numbers and timestamps so the receiver's jitter buffer can reorder out-of-sequence UDP packets.
Narrowband Audio Encoding (ITU-T Recommendation G.711): PSTN audio is typically encoded using G.711 pulse code modulation (PCM) at an 8,000 Hz (8 kHz) sampling rate and 64 kbit/s bitrate, using either mu-law (PCMU, standard in North America and Japan) or A-law (PCMA, standard in India, Europe, and international routes). G.711 narrowband audio is typically limited to roughly 300 to 3,400 Hz, well below the frequency range that carries some high-frequency fricative distinctions (such as "s" vs. "f"), which can make narrowband audio harder for speech recognizers trained primarily on wideband audio.
Full-Duplex WebSocket Bridging (IETF RFC 6455): RFC 6455 defines the WebSocket protocol's persistent, bidirectional connection and message framing. Media gateways use this transport to bridge the RTP stream to your orchestrator; the specific chunk size and control-event format, such as 20 ms audio chunks and media/mark/clear events, are defined by the telephony provider's media-streaming product, such as Twilio's Media Streams WebSocket messages.

End-to-end telephony voice AI pipeline bridging 8 kHz G.711 SIP/RTP phone audio over a full-duplex WebSocket through local Silero VAD, streaming STT, clause-chunked LLM generation, and streaming TTS.

In a sequential architecture, total turnaround delay is the sum of the full completion time of every stage:

Sequential turn delay = T_Endpointing + T_Full_STT + T_Full_LLM + T_Full_TTS + T_Network

In a pipelined streaming architecture, each stage emits incremental frames to the next stage before the upstream task finishes. STT transcribes while the user speaks; the LLM begins generating tokens as soon as the turn endpoint triggers; TTS begins synthesizing audio as soon as the first clause forms; and the carrier gateway plays the first G.711 audio packets while the LLM is still generating the second sentence.

Pipelined turn delay = T_Endpointing + T_STT_Finalization + T_LLM_TTFT_First_Clause + T_TTS_TTFB + T_Network

Conversational and ITU-T G.114 latency benchmarks compared against sequential and pipelined voice AI turn stages, using sourced millisecond thresholds and qualitative streaming labels.

2. Dual-Signal Turn Detection: Combining Audio VAD with Word-Timing Endpointing

Determining when a caller has finished speaking is one of the hardest trade-offs in voice engineering:

If your endpointing threshold is too short, the agent interrupts the caller every time they pause to breathe, recall an order number, or switch languages mid-sentence.
If your endpointing threshold is too long, every single turn suffers from an artificial silence penalty before the LLM is even invoked.

Relying on a single silence timer fails on real phone calls because phone environments are noisy. Autorickshaw traffic, office chatter, ceiling fans, or carrier line hum can keep an audio-only Voice Activity Detector open indefinitely, preventing the agent from ever responding.

Layer 1: Fixed-Window Acoustic VAD

On the audio ingest path, lightweight neural voice activity detectors evaluate raw audio frames to distinguish human speech from stationary background noise. For example, Silero VAD (snakers4/silero-vad) processes audio in strict 32-millisecond windows: 256 samples per frame at an 8 kHz telephony sample rate (256 / 8000 = 0.032 s) or 512 samples per frame at 16 kHz (512 / 16000 = 0.032 s). Running an acoustic VAD on incoming frames gives your orchestrator an immediate local signal when speech starts (speech_start) and when acoustic energy drops.

Layer 2: Combining `endpointing` and `utterance_end_ms` in Streaming STT

When streaming audio over WebSockets to a speech recognizer such as Deepgram, one workable design combines two complementary mechanisms documented in the Deepgram Streaming STT specification:

VAD-Based Endpointing (endpointing): Deepgram's endpointing parameter monitors the incoming audio waveform for a transition from speech to silence (configurable in milliseconds; while the API default is 10 ms, conversational agents commonly configure 300 to 500 ms so brief mid-sentence pauses do not split a thought). When that silence window elapses, the recognizer locks the segment and emits a transcript response with speech_final: true.
Word-Timing Gap Detection (utterance_end_ms + interim_results): Because non-speech background noise on a phone line can prevent audio VAD from detecting silence, Deepgram's utterance_end_ms parameter operates on word timestamps rather than raw audio silence. It requires interim_results=true and measures the time elapsed since the last finalized word. If no new finalized word arrives within the configured window (for example, 1000 ms), it emits an {"type": "UtteranceEnd"} control event over the WebSocket, even if background noise is still present, and even if the caller resumes speaking after a brief pause, so treat it as a fallback signal rather than a guaranteed turn boundary.

Mechanism	Signal Evaluated	Primary Strength	Failure Mode on Phone Calls	Role in Pipeline
Acoustic VAD (`endpointing`)	Raw audio waveform energy and silence	Fast detection on quiet phone lines	Loud background noise can keep the stream open (`speech_final` never fires)	Fast-path turn trigger (one common design)
Word-Timing Gap (`utterance_end_ms`)	Time gap since last finalized word in `interim_results`	Ignores non-speech background noise	Needs a slightly longer timeout window to avoid premature firing	Fallback turn trigger (one common design)

In your orchestrator, accumulate finalized transcript segments as they arrive and use speech_final: true as the signal that the accumulated segments form a complete turn ready to dispatch to the LLM. A single utterance can span multiple is_final: true responses, so dispatching only the segment attached to the speech_final event can clip earlier words from the turn. Use UtteranceEnd as the safety net whenever speech_final has not fired due to line noise.

3. Clause-Boundary Token Streaming Without Breaking TTS Prosody

Once the turn transcript is dispatched to the LLM via a streaming API (such as Anthropic's Server-Sent Events streaming messages or OpenAI's streaming chat completions), tokens arrive incrementally.

A common mistake in early voice AI implementations is splitting the LLM token stream with a naive regular expression like /[.,?!]/ and sending every fragment straight to TTS. In production, naive punctuation splitting causes three problems:

Abbreviation and numeric clipping: Splitting on every period breaks sentences in half on honorifics (Dr., Mr., Mrs.), currency and measurements (Rs. 500, $19.99, 3.5%), or initials (U.S., e.g.).
Starved TTS prosody: As documented in the ElevenLabs WebSocket Streaming TTS reference, the synthesizer buffers incoming text against a chunk_length_schedule (an array of cumulative character thresholds such as [120, 160, 250, 290]) before generating audio frames unless forced to flush. In this design, sending two-word fragments with forced flushes trades away the forward text context the model uses for pitch contour and word stress, producing more robotic, choppy intonation.
Code-mixed punctuation: In Indian phone conversations where callers and agents speak Hinglish (Hindi-English code-mixing) or Devanagari alongside Latin script, sentence boundaries may use the Devanagari danda (।) in addition to standard ASCII punctuation.

Implementation: Abbreviation-Safe Clause Buffer

Instead of splitting on raw punctuation characters, buffer incoming LLM tokens until both a minimum character threshold is met and a safe syntactic boundary is followed by whitespace:

const COMMON_ABBREVIATIONS = new Set([
  'mr', 'mrs', 'ms', 'dr', 'prof', 'sr', 'jr',
  'rs', 'st', 'no', 'vs', 'etc', 'eg', 'ie',
]);

export interface ClauseChunkerOptions {
  minClauseChars?: number;
  softSplitChars?: number;
  onClause: (clause: string) => void;
}

export class AbbreviationSafeClauseChunker {
  private buffer = '';
  private readonly minClauseChars: number;
  private readonly softSplitChars: number;
  private readonly onClause: (clause: string) => void;

  constructor(options: ClauseChunkerOptions) {
    this.minClauseChars = options.minClauseChars ?? 45;
    this.softSplitChars = options.softSplitChars ?? 110;
    this.onClause = options.onClause;
  }

  public pushToken(token: string, signal?: AbortSignal): void {
    if (signal?.aborted) return;
    this.buffer += token;
    this.extractClauses(signal);
  }

  public flush(signal?: AbortSignal): void {
    if (signal?.aborted) {
      this.buffer = '';
      return;
    }
    const remaining = this.buffer.trim();
    this.buffer = '';
    if (remaining.length > 0) {
      this.onClause(remaining);
    }
  }

  public reset(): void {
    this.buffer = '';
  }

  private extractClauses(signal?: AbortSignal): void {
    // Match terminal punctuation (. ? ! or Devanagari danda ।) or soft clause
    // punctuation (, ; :) followed by whitespace
    const boundaryRegex = /([.?!।]|[,;:])(\s+)/g;
    let match: RegExpExecArray | null;

    while ((match = boundaryRegex.exec(this.buffer)) !== null) {
      if (signal?.aborted) {
        this.buffer = '';
        return;
      }

      const punct = match[1];
      const splitIndex = match.index + punct.length;
      const candidate = this.buffer.slice(0, splitIndex).trim();

      if (candidate.length < this.minClauseChars) {
        continue;
      }

      const isTerminal = /[.?!।]/.test(punct);
      if (!isTerminal && candidate.length < this.softSplitChars) {
        // Hold commas/semicolons unless the clause is already long enough
        continue;
      }

      if (punct === '.' && this.isAbbreviationOrDecimal(candidate)) {
        continue;
      }

      this.onClause(candidate);
      this.buffer = this.buffer.slice(splitIndex + match[2].length);
      boundaryRegex.lastIndex = 0;
    }
  }

  private isAbbreviationOrDecimal(candidate: string): boolean {
    // Decimals (e.g. 19.99, 3.5%) do not trigger splits because boundaryRegex
    // requires trailing whitespace. For candidate ending with a period:
    // Check for numbered list items / section numbers (e.g. "Step 1.", "2.")
    if (/\b\d+\.$/.test(candidate)) {
      return true;
    }
    // Check for dotted abbreviations (e.g. "e.g.", "i.e.", "U.S.") or titles ("Dr.", "Rs.")
    const wordMatch = candidate.match(/([A-Za-z.]+)\.$/);
    if (wordMatch) {
      const normalized = wordMatch[1].toLowerCase().replace(/\./g, '');
      if (normalized.length === 1 || COMMON_ABBREVIATIONS.has(normalized)) {
        return true;
      }
    }
    return false;
  }
}

This pattern lets you dispatch the opening sentence to the TTS WebSocket as soon as a complete, prosodically stable thought is ready, while longer sentences automatically split at commas or semicolons once softSplitChars is reached.

4. Full-Stack Barge-In: Flushing Media Gateways and Truncating Context

In a natural phone call, callers interrupt the agent when they already know the answer, when the agent misunderstood their request, or when they want to skip a long explanation.

Most developer tutorials treat interruption handling (barge-in) as calling AbortController.abort() on the active LLM request. On a real phone call, aborting the LLM is only half of the job.

Why Backend-Only Cancellation Fails on Phone Calls

When your TTS engine synthesizes audio faster than real-time playback, it pushes several seconds of G.711 audio packets into the telephony gateway's outbound buffer. If the caller interrupts and you only cancel the LLM and TTS tasks on your server, the telephony gateway will continue playing the buffered audio for several more seconds before silence occurs.

Furthermore, if the LLM had already streamed three sentences before the caller interrupted during sentence one, saving all three sentences in the assistant's conversation history causes state drift: on the next turn, the LLM believes the caller heard facts in sentences two and three that were never actually played over the phone line.

The Four-Step Barge-In Protocol

Using bidirectional telephony WebSocket messages (such as the media, mark, and clear events specified in Twilio's Media Streams WebSocket documentation), a production orchestrator handles barge-in across four coordinated steps:

Tag outbound audio segments with mark checkpoints: Every time you stream synthesized G.711 audio for a clause (clause_0, clause_1, clause_2), send a trailing mark message over the gateway WebSocket. The gateway echoes that mark event back to your server once the associated buffered media has completed playback, but note that sending a clear command can also cause outstanding marks to be returned immediately, so a mark echo alone does not always mean the audio played to completion.
Filter passive backchannels from real interruptions: Callers frequently say brief acknowledgments ("uh-huh", "okay", "right", or "haan" and "accha" on Indian phone calls) while listening. If your acoustic VAD mutes playback on every 100 ms sound burst, a polite listener will constantly stall the agent. Require either sustained speech energy across consecutive Silero VAD frames or a multi-word interim STT transcript before firing a full barge-in.
Send a clear command to flush the gateway buffer: The instant a true interruption is confirmed, send {"event": "clear", "streamSid": "..."} over the telephony WebSocket while triggering your local AbortController. The gateway immediately purges its outbound audio buffer so the phone speaker goes silent.
Truncate assistant history to confirmed playback: When recording the interrupted assistant turn in your LLM conversation context, track each dispatched clause's state as pending, playing, played, or cleared. Include only clauses whose mark checkpoint was echoed back as played before the interruption; do not assume the clause that was actively playing when clear was sent was fully heard, since a clear command can cause the gateway to return a mark for audio it never finished playing.

Full-stack barge-in sequence showing outbound mark tracking, VAD and interim STT interruption confirmation, gateway buffer clearing, LLM/TTS cancellation, and conversation history truncation.

type ClauseState = 'pending' | 'playing' | 'played' | 'cleared';

interface ClauseCheckpoint {
  markId: string;
  text: string;
  state: ClauseState;
}

export class TurnPlaybackTracker {
  private checkpoints: ClauseCheckpoint[] = [];

  public registerClause(markId: string, text: string): void {
    this.checkpoints.push({ markId, text, state: 'pending' });
  }

  public onClauseDispatchedToGateway(markId: string): void {
    const item = this.checkpoints.find((c) => c.markId === markId);
    if (item && item.state === 'pending') {
      item.state = 'playing';
    }
  }

  public onGatewayMarkEchoed(markId: string): void {
    const item = this.checkpoints.find((c) => c.markId === markId);
    // A clear command can echo marks for audio that was never actually
    // played, so a mark already flagged `cleared` must not flip to `played`.
    if (item && item.state !== 'cleared') {
      item.state = 'played';
    }
  }

  public onGatewayCleared(): void {
    for (const item of this.checkpoints) {
      if (item.state === 'pending' || item.state === 'playing') {
        item.state = 'cleared';
      }
    }
  }

  public getSpokenTextOnInterrupt(): string {
    // Only clauses confirmed `played` are known to have reached the caller;
    // a clause that was merely `playing` when `clear` fired is not included
    // because how much of it was actually heard is unknown.
    return this.checkpoints
      .filter((c) => c.state === 'played')
      .map((c) => c.text)
      .join(' ');
  }
}

5. Decoupling Mid-Call Tool and Webhook Execution from the Audio Loop

Voice agents on real phone calls rarely exist just to chat. They check appointment slots, verify order status, look up CRM records, or trigger payment links via external webhooks and structured LLM function calling.

If your voice orchestrator awaits a synchronous HTTP request to an external CRM or booking API inside the turn loop before emitting any audio, the caller hears silence for the entire duration of the third-party API call.

To keep the conversation alive during external I/O:

Emit an immediate verbal bridge before awaiting the tool: Configure your system prompt or tool router so that when the model emits a tool call block, the orchestrator immediately dispatches a brief, context-appropriate acknowledgment clause to TTS ("Let me check your booking details right now.") while the HTTP request executes concurrently.
Bind tool execution to the turn's AbortSignal: Callers often add extra details while the agent is looking something up ("Actually, check under my other phone number..."). Pass the turn's AbortSignal into fetch() for read-only lookups so stale requests are cancelled cleanly if the caller barges in, and use idempotency keys on state-mutating webhooks so retries never double-book an appointment.
Enforce strict timeouts with conversational fallbacks: Never allow an external webhook to hang a phone call indefinitely. Wrap every tool invocation in a hard timeout budget so that if a downstream CRM is slow or unreachable, the orchestrator catches the timeout and instructs the agent to offer an alternative path (such as escalating to a human representative or following up via SMS).

6. Architectural Checklist for Production Voice Pipelines

Before routing live phone traffic to a voice AI pipeline, verify each stage of your stack against these engineering checks:

Pipeline Layer	Engineering Requirement	Why It Matters on Phone Calls
Telephony & Codec	Native 8 kHz G.711 (`PCMU` / `PCMA`) handling and STT models tuned for narrowband audio, where available (ITU-T G.711)	Helps limit accuracy loss on narrowband carrier audio and can reduce unnecessary transcoding hops
Turn Detection	Dual-signal endpointing: acoustic VAD (`endpointing`) paired with word-timing gap detection (`utterance_end_ms`)	Prevents noisy phone lines from keeping the turn open forever while respecting natural pauses
LLM-to-TTS Handoff	Abbreviation-safe clause chunker with minimum character thresholds	Prevents clipped speech on `Dr.`, `Rs.`, and decimals while feeding TTS enough context for natural prosody
Barge-In & State	Gateway `clear` message dispatch + `mark`-based context truncation (Twilio Media Streams)	Stops phone playback immediately on interruption and keeps LLM memory synced with what the caller heard
Tool & Webhook I/O	Asynchronous tool execution with verbal acknowledgment and hard timeouts	Prevents slow external CRMs or APIs from creating dead air mid-call
Multilingual & Code-Mixing	Support for mid-call language switching and code-mixed speech (such as Hinglish)	Callers in multilingual markets frequently switch between English and local languages within a single call

Building Without Wiring Every Protocol Layer Yourself

Putting these pieces together from scratch means maintaining SIP/RTP media bridges, WebSocket state machines, VAD frame buffers, abbreviation-aware token chunkers, gateway clear/mark synchronization, and post-call webhook pipelines across multiple model vendors.

If you want to focus on your agent's logic and tools rather than maintaining the underlying voice infrastructure, Rymi provides a unified voice AI platform behind one API for STT, LLM, TTS, and telephony, with voices for English and Hindi, including Hinglish. You can bring your own carrier numbers (Twilio, Plivo, Vonage, or Telnyx) and choose between tuned managed stacks or custom bring-your-own-key stacks. Pricing is straightforward: Managed from $0.05/min, or BYO providers at cost plus $0.02/min runtime.

Explore how Rymi compares to other platforms on our comparison page, read the API documentation, or create a free account to test an agent directly from your browser.