The fix is a pipelined architecture where every downstream stage starts working on partial output before the upstream stage finishes. Concretely, that means wiring together five pieces end to end:

1. Bridging narrowband PSTN/SIP telephony streams to full-duplex WebSockets.
2. Combining local audio Voice Activity Detection (VAD) with word-timing endpointing so noisy phone lines do not break turn detection.
3. Streaming LLM tokens into TTS at safe syntactic clause boundaries without clipping prosody or choking on abbreviations.
4. Executing full-stack barge-in by flushing telephony gateway playback buffers and truncating conversation history to what the caller actually heard.
5. Decoupling synchronous webhook and CRM tool calls from the audio loop.

In human conversation, turn-taking is remarkably fast. A cross-linguistic study by Stivers et al. (2009) in the Proceedings of the National Academy of Sciences found that across ten languages, the cross-language mean response offset was about 208 milliseconds, while the overall modal (most common) offset was 0 milliseconds, meaning the single most frequent pattern was the next speaker starting at almost the same instant the current speaker finished, not after a gap. Because preparing a single spoken word takes roughly 600 milliseconds before articulation begins (an estimate from picture-naming studies, reviewed by Levinson and Torreira (2015) in Frontiers in Psychology), human listeners do not wait for silence to start thinking; they overlap comprehension with response formulation while the other person is still speaking.

In telecommunications network planning, ITU-T Recommendation G.114 recommends that one-way transmission delay for general network planning should not exceed 400 milliseconds, while cautioning that highly interactive tasks can be perceptibly affected by much lower delays than that ceiling.

When you place an AI pipeline in the middle of a phone call, every millisecond of silence after the caller stops speaking adds to that turn-taking gap. If your backend waits for Speech-to-Text (STT) to finalize a full paragraph, then waits for a Large Language Model (LLM) to generate a complete answer, and finally waits for Text-to-Speech (TTS) to render a full audio file before streaming bytes back to the carrier gateway, the caller hears dead air and assumes the call dropped.

At Rymi, we build voice AI infrastructure that exposes one API for STT, LLM, TTS, and telephony, with voices for English and Hindi, including Hinglish. While our earlier guide on production voice AI agents covered provider trade-offs and failure fallbacks, this deep dive focuses on the protocol and orchestration mechanics behind the five pieces above.

1. The Telephony-to-Model Pipeline: What Happens on a Phone Call

A browser-based voice demo captures wideband microphone audio (typically 16 kHz or 48 kHz) over WebRTC. A standard public switched telephone network (PSTN) call operates under much stricter physical and protocol constraints:

1. Call Signaling (IETF RFC 3261): Session Initiation Protocol (SIP) negotiates the call session, codecs, and media endpoints between the carrier and your Session Border Controller (SBC) or telephony provider (such as Twilio, Plivo, Vonage, or Telnyx).
2. Media Transport (IETF RFC 3550): Once the call connects, audio frames travel over the Real-time Transport Protocol (RTP), carrying sequence numbers and timestamps so the receiver's jitter buffer can reorder out-of-sequence UDP packets.
3. Narrowband Audio Encoding (ITU-T Recommendation G.711): PSTN audio is typically encoded using G.711 pulse code modulation (PCM) at an 8,000 Hz (8 kHz) sampling rate and 64 kbit/s bitrate, using either mu-law (PCMU, standard in North America and Japan) or A-law (PCMA, standard in India, Europe, and international routes). G.711 narrowband audio is typically limited to roughly 300 to 3,400 Hz, well below the frequency range that carries some high-frequency fricative distinctions (such as "s" vs. "f"), which can make narrowband audio harder for speech recognizers trained primarily on wideband audio.
4. Full-Duplex WebSocket Bridging (IETF RFC 6455): RFC 6455 defines the WebSocket protocol's persistent, bidirectional connection and message framing. Media gateways use this transport to bridge the RTP stream to your orchestrator; the specific chunk size and control-event format, such as 20 ms audio chunks and media/mark/clear events, are defined by the telephony provider's media-streaming product, such as Twilio's Media Streams WebSocket messages.

End-to-end telephony voice AI pipeline bridging 8 kHz G.711 SIP/RTP phone audio over a full-duplex WebSocket through local Silero VAD, streaming STT, clause-chunked LLM generation, and streaming TTS.

In a sequential architecture, total turnaround delay is the sum of the full completion time of every stage:

Sequential turn delay = T_Endpointing + T_Full_STT + T_Full_LLM + T_Full_TTS + T_Network

In a pipelined streaming architecture, each stage emits incremental frames to the next stage before the upstream task finishes. STT transcribes while the user speaks; the LLM begins generating tokens as soon as the turn endpoint triggers; TTS begins synthesizing audio as soon as the first clause forms; and the carrier gateway plays the first G.711 audio packets while the LLM is still generating the second sentence.

Pipelined turn delay = T_Endpointing + T_STT_Finalization + T_LLM_TTFT_First_Clause + T_TTS_TTFB + T_Network

Conversational and ITU-T G.114 latency benchmarks compared against sequential and pipelined voice AI turn stages, using sourced millisecond thresholds and qualitative streaming labels.

2. Dual-Signal Turn Detection: Combining Audio VAD with Word-Timing Endpointing

Determining when a caller has finished speaking is one of the hardest trade-offs in voice engineering:

• If your endpointing threshold is too short, the agent interrupts the caller every time they pause to breathe, recall an order number, or switch languages mid-sentence.
• If your endpointing threshold is too long, every single turn suffers from an artificial silence penalty before the LLM is even invoked.

Relying on a single silence timer fails on real phone calls because phone environments are noisy. Autorickshaw traffic, office chatter, ceiling fans, or carrier line hum can keep an audio-only Voice Activity Detector open indefinitely, preventing the agent from ever responding.

Layer 1: Fixed-Window Acoustic VAD

On the audio ingest path, lightweight neural voice activity detectors evaluate raw audio frames to distinguish human speech from stationary background noise. For example, Silero VAD (snakers4/silero-vad) processes audio in strict 32-millisecond windows: 256 samples per frame at an 8 kHz telephony sample rate (256 / 8000 = 0.032 s) or 512 samples per frame at 16 kHz (512 / 16000 = 0.032 s). Running an acoustic VAD on incoming frames gives your orchestrator an immediate local signal when speech starts (speech_start) and when acoustic energy drops.

Layer 2: Combining endpointing and utterance_end_ms in Streaming STT

When streaming audio over WebSockets to a speech recognizer such as Deepgram, one workable design combines two complementary mechanisms documented in the Deepgram Streaming STT specification:

1. VAD-Based Endpointing (endpointing): Deepgram's endpointing parameter monitors the incoming audio waveform for a transition from speech to silence (configurable in milliseconds; while the API default is 10 ms, conversational agents commonly configure 300 to 500 ms so brief mid-sentence pauses do not split a thought). When that silence window elapses, the recognizer locks the segment and emits a transcript response with speech_final: true.
2. Word-Timing Gap Detection (utterance_end_ms + interim_results): Because non-speech background noise on a phone line can prevent audio VAD from detecting silence, Deepgram's utterance_end_ms parameter operates on word timestamps rather than raw audio silence. It requires interim_results=true and measures the time elapsed since the last finalized word. If no new finalized word arrives within the configured window (for example, 1000 ms), it emits an {"type": "UtteranceEnd"} control event over the WebSocket, even if background noise is still present, and even if the caller resumes speaking after a brief pause, so treat it as a fallback signal rather than a guaranteed turn boundary.

In your orchestrator, accumulate finalized transcript segments as they arrive and use speech_final: true as the signal that the accumulated segments form a complete turn ready to dispatch to the LLM. A single utterance can span multiple is_final: true responses, so dispatching only the segment attached to the speech_final event can clip earlier words from the turn. Use UtteranceEnd as the safety net whenever speech_final has not fired due to line noise.

3. Clause-Boundary Token Streaming Without Breaking TTS Prosody

Once the turn transcript is dispatched to the LLM via a streaming API (such as Anthropic's Server-Sent Events streaming messages or OpenAI's streaming chat completions), tokens arrive incrementally.

A common mistake in early voice AI implementations is splitting the LLM token stream with a naive regular expression like /[.,?!]/ and sending every fragment straight to TTS. In production, naive punctuation splitting causes three problems:

1. Abbreviation and numeric clipping: Splitting on every period breaks sentences in half on honorifics (Dr., Mr., Mrs.), currency and measurements (Rs. 500, $19.99, 3.5%), or initials (U.S., e.g.).
2. Starved TTS prosody: As documented in the ElevenLabs WebSocket Streaming TTS reference, the synthesizer buffers incoming text against a chunk_length_schedule (an array of cumulative character thresholds such as [120, 160, 250, 290]) before generating audio frames unless forced to flush. In this design, sending two-word fragments with forced flushes trades away the forward text context the model uses for pitch contour and word stress, producing more robotic, choppy intonation.
3. Code-mixed punctuation: In Indian phone conversations where callers and agents speak Hinglish (Hindi-English code-mixing) or Devanagari alongside Latin script, sentence boundaries may use the Devanagari danda (।) in addition to standard ASCII punctuation.

Implementation: Abbreviation-Safe Clause Buffer

Instead of splitting on raw punctuation characters, buffer incoming LLM tokens until both a minimum character threshold is met and a safe syntactic boundary is followed by whitespace:

const COMMON_ABBREVIATIONS = new Set([
  'mr', 'mrs', 'ms', 'dr', 'prof', 'sr', 'jr',
  'rs', 'st', 'no', 'vs', 'etc', 'eg', 'ie',
]);

export interface ClauseChunkerOptions {
  minClauseChars?: number;
  softSplitChars?: number;
  onClause: (clause: string) => void;
}

export class AbbreviationSafeClauseChunker {
  private buffer = '';
  private readonly minClauseChars: number;
  private readonly softSplitChars: number;
  private readonly onClause: (clause: string) => void;

  constructor(options: ClauseChunkerOptions) {
    this.minClauseChars = options.minClauseChars ?? 45;
    this.softSplitChars = options.softSplitChars ?? 110;
    this.onClause = options.onClause;
  }

  public pushToken(token: string, signal?: AbortSignal): void {
    if (signal?.aborted) return;
    this.buffer += token;
    this.extractClauses(signal);
  }

  public flush(signal?: AbortSignal): void {
    if (signal?.aborted) {
      this.buffer = '';
      return;
    }
    const remaining = this.buffer.trim();
    this.buffer = '';
    if (remaining.length > 0) {
      this.onClause(remaining);
    }
  }

  public reset(): void {
    this.buffer = '';
  }

  private extractClauses(signal?: AbortSignal): void {
    // Match terminal punctuation (. ? ! or Devanagari danda ।) or soft clause
    // punctuation (, ; :) followed by whitespace
    const boundaryRegex = /([.?!।]|[,;:])(\s+)/g;
    let match: RegExpExecArray | null;

    while ((match = boundaryRegex.exec(this.buffer)) !== null) {
      if (signal?.aborted) {
        this.buffer = '';
        return;
      }

      const punct = match[1];
      const splitIndex = match.index + punct.length;
      const candidate = this.buffer.slice(0, splitIndex).trim();

      if (candidate.length < this.minClauseChars) {
        continue;
      }

      const isTerminal = /[.?!।]/.test(punct);
      if (!isTerminal && candidate.length < this.softSplitChars) {
        // Hold commas/semicolons unless the clause is already long enough
        continue;
      }

      if (punct === '.' && this.isAbbreviationOrDecimal(candidate)) {
        continue;
      }

      this.onClause(candidate);
      this.buffer = this.buffer.slice(splitIndex + match[2].length);
      boundaryRegex.lastIndex = 0;
    }
  }

  private isAbbreviationOrDecimal(candidate: string): boolean {
    // Decimals (e.g. 19.99, 3.5%) do not trigger splits because boundaryRegex
    // requires trailing whitespace. For candidate ending with a period:
    // Check for numbered list items / section numbers (e.g. "Step 1.", "2.")
    if (/\b\d+\.$/.test(candidate)) {
      return true;
    }
    // Check for dotted abbreviations (e.g. "e.g.", "i.e.", "U.S.") or titles ("Dr.", "Rs.")
    const wordMatch = candidate.match(/([A-Za-z.]+)\.$/);
    if (wordMatch) {
      const normalized = wordMatch[1].toLowerCase().replace(/\./g, '');
      if (normalized.length === 1 || COMMON_ABBREVIATIONS.has(normalized)) {
        return true;
      }
    }
    return false;
  }
}

This pattern lets you dispatch the opening sentence to the TTS WebSocket as soon as a complete, prosodically stable thought is ready, while longer sentences automatically split at commas or semicolons once softSplitChars is reached.

4. Full-Stack Barge-In: Flushing Media Gateways and Truncating Context

In a natural phone call, callers interrupt the agent when they already know the answer, when the agent misunderstood their request, or when they want to skip a long explanation.

Most developer tutorials treat interruption handling (barge-in) as calling AbortController.abort() on the active LLM request. On a real phone call, aborting the LLM is only half of the job.

Why Backend-Only Cancellation Fails on Phone Calls

When your TTS engine synthesizes audio faster than real-time playback, it pushes several seconds of G.711 audio packets into the telephony gateway's outbound buffer. If the caller interrupts and you only cancel the LLM and TTS tasks on your server, the telephony gateway will continue playing the buffered audio for several more seconds before silence occurs.

Furthermore, if the LLM had already streamed three sentences before the caller interrupted during sentence one, saving all three sentences in the assistant's conversation history causes state drift: on the next turn, the LLM believes the caller heard facts in sentences two and three that were never actually played over the phone line.

The Four-Step Barge-In Protocol

Using bidirectional telephony WebSocket messages (such as the media, mark, and clear events specified in Twilio's Media Streams WebSocket documentation), a production orchestrator handles barge-in across four coordinated steps:

1. Tag outbound audio segments with mark checkpoints: Every time you stream synthesized G.711 audio for a clause (clause_0, clause_1, clause_2), send a trailing mark message over the gateway WebSocket. The gateway echoes that mark event back to your server once the associated buffered media has completed playback, but note that sending a clear command can also cause outstanding marks to be returned immediately, so a mark echo alone does not always mean the audio played to completion.
2. Filter passive backchannels from real interruptions: Callers frequently say brief acknowledgments ("uh-huh", "okay", "right", or "haan" and "accha" on Indian phone calls) while listening. If your acoustic VAD mutes playback on every 100 ms sound burst, a polite listener will constantly stall the agent. Require either sustained speech energy across consecutive Silero VAD frames or a multi-word interim STT transcript before firing a full barge-in.
3. Send a clear command to flush the gateway buffer: The instant a true interruption is confirmed, send {"event": "clear", "streamSid": "..."} over the telephony WebSocket while triggering your local AbortController. The gateway immediately purges its outbound audio buffer so the phone speaker goes silent.
4. Truncate assistant history to confirmed playback: When recording the interrupted assistant turn in your LLM conversation context, track each dispatched clause's state as pending, playing, played, or cleared. Include only clauses whose mark checkpoint was echoed back as played before the interruption; do not assume the clause that was actively playing when clear was sent was fully heard, since a clear command can cause the gateway to return a mark for audio it never finished playing.

Full-stack barge-in sequence showing outbound mark tracking, VAD and interim STT interruption confirmation, gateway buffer clearing, LLM/TTS cancellation, and conversation history truncation.

type ClauseState = 'pending' | 'playing' | 'played' | 'cleared';

interface ClauseCheckpoint {
  markId: string;
  text: string;
  state: ClauseState;
}

export class TurnPlaybackTracker {
  private checkpoints: ClauseCheckpoint[] = [];

  public registerClause(markId: string, text: string): void {
    this.checkpoints.push({ markId, text, state: 'pending' });
  }

  public onClauseDispatchedToGateway(markId: string): void {
    const item = this.checkpoints.find((c) => c.markId === markId);
    if (item && item.state === 'pending') {
      item.state = 'playing';
    }
  }

  public onGatewayMarkEchoed(markId: string): void {
    const item = this.checkpoints.find((c) => c.markId === markId);
    // A clear command can echo marks for audio that was never actually
    // played, so a mark already flagged `cleared` must not flip to `played`.
    if (item && item.state !== 'cleared') {
      item.state = 'played';
    }
  }

  public onGatewayCleared(): void {
    for (const item of this.checkpoints) {
      if (item.state === 'pending' || item.state === 'playing') {
        item.state = 'cleared';
      }
    }
  }

  public getSpokenTextOnInterrupt(): string {
    // Only clauses confirmed `played` are known to have reached the caller;
    // a clause that was merely `playing` when `clear` fired is not included
    // because how much of it was actually heard is unknown.
    return this.checkpoints
      .filter((c) => c.state === 'played')
      .map((c) => c.text)
      .join(' ');
  }
}

5. Decoupling Mid-Call Tool and Webhook Execution from the Audio Loop

Voice agents on real phone calls rarely exist just to chat. They check appointment slots, verify order status, look up CRM records, or trigger payment links via external webhooks and structured LLM function calling.

If your voice orchestrator awaits a synchronous HTTP request to an external CRM or booking API inside the turn loop before emitting any audio, the caller hears silence for the entire duration of the third-party API call.

To keep the conversation alive during external I/O:

1. Emit an immediate verbal bridge before awaiting the tool: Configure your system prompt or tool router so that when the model emits a tool call block, the orchestrator immediately dispatches a brief, context-appropriate acknowledgment clause to TTS ("Let me check your booking details right now.") while the HTTP request executes concurrently.
2. Bind tool execution to the turn's AbortSignal: Callers often add extra details while the agent is looking something up ("Actually, check under my other phone number..."). Pass the turn's AbortSignal into fetch() for read-only lookups so stale requests are cancelled cleanly if the caller barges in, and use idempotency keys on state-mutating webhooks so retries never double-book an appointment.
3. Enforce strict timeouts with conversational fallbacks: Never allow an external webhook to hang a phone call indefinitely. Wrap every tool invocation in a hard timeout budget so that if a downstream CRM is slow or unreachable, the orchestrator catches the timeout and instructs the agent to offer an alternative path (such as escalating to a human representative or following up via SMS).

6. Architectural Checklist for Production Voice Pipelines

Before routing live phone traffic to a voice AI pipeline, verify each stage of your stack against these engineering checks:

Building Without Wiring Every Protocol Layer Yourself

Putting these pieces together from scratch means maintaining SIP/RTP media bridges, WebSocket state machines, VAD frame buffers, abbreviation-aware token chunkers, gateway clear/mark synchronization, and post-call webhook pipelines across multiple model vendors.

If you want to focus on your agent's logic and tools rather than maintaining the underlying voice infrastructure, Rymi provides a unified voice AI platform behind one API for STT, LLM, TTS, and telephony, with voices for English and Hindi, including Hinglish. You can bring your own carrier numbers (Twilio, Plivo, Vonage, or Telnyx) and choose between tuned managed stacks or custom bring-your-own-key stacks. Pricing is straightforward: Managed from $0.05/min, or BYO providers at cost plus $0.02/min runtime.

Explore how Rymi compares to other platforms on our comparison page, read the API documentation, or create a free account to test an agent directly from your browser.

Building a voice AI agent sounds straightforward. You pipe audio into a speech model, feed the text to an LLM, convert the response back to audio, and play it to the caller. Simple, right?

It is, until you're in production and a caller is waiting. Then 200ms starts feeling like an eternity, a single transcription error cascades through your entire context window, and a TTS hiccup sounds like a frozen phone line. The LLM was always the easy part. The hard part is systems integration under real-time constraints.

This post covers the architectural lessons we've accumulated building Rymi, a Voice AI infrastructure platform. None of this is theoretical. It's what actually breaks in production.

The architecture, plainly

Every production voice agent is the same five-stage pipeline:

Phone / SIP → STT → Context + LLM → TTS → Audio out

The problem isn't any individual component. The problem is that each of them must fit inside roughly one second total, the threshold after which a human caller notices a delay and starts doubting whether the system is working.

The 1-second budget

Before writing a single line of code, budget your latency like a strict expense account:

Miss any one of these targets and the entire experience degrades. There's no single component to optimize. You're constantly balancing the whole system.

Speech-to-text: speed beats accuracy

This is counter-intuitive. You'd assume you want the most accurate transcription possible. In practice, a 97%-accurate STT that takes 800ms feels worse than a 93%-accurate STT that takes 300ms. Users perceive responsiveness, not accuracy.

Provider figures as of April 2026, from each vendor's published documentation. Latency and pricing move; check the vendor before you commit.

For phone calls, Deepgram with streaming is currently the strongest option. It supports interim_results and VAD (Voice Activity Detection) events, which means you can show the caller an acknowledgment signal while waiting for the final transcript. This alone shaves 200ms off perceived latency.

Phone audio is not regular audio. Phone calls are typically 8kHz narrowband, compressed, and acoustically degraded. Don't use a generic STT model trained on clear speech. Deepgram's "phone" model is specifically trained for this. The accuracy delta is significant.

LLM inference: stream, don't batch

Standard LLM API calls have 500ms+ latency before you receive the first token. If you wait for a complete response before starting TTS, you've already consumed more than half your budget on one component.

The solution is streaming at every stage: start TTS generation the moment you have the first complete sentence from the LLM, not when the full response is done:

// Start TTS as soon as a sentence boundary arrives
for await (const chunk of llmStream) {
  buffer += chunk;

  // Detect sentence boundary
  if (/[.!?]\s*$/.test(buffer)) {
    ttsQueue.push(buffer);
    buffer = '';
    if (!ttsPlaying) playNextChunk();
  }
}

The audio for the first sentence is already playing by the time the LLM finishes generating the second. The caller experiences a continuous response, not a loading pause.

Model selection

Model figures as of April 2026. Newer tiers have since shipped. See the model picker for what Rymi runs today.

For voice agents handling qualification, booking, or FAQ tasks, Claude 3 Haiku consistently fits the latency budget while producing natural-sounding responses. Save the heavier models for post-call analysis, not live calls.

Context management: the part most teams under-build

Raw transcribed text has no context. "That works for me" means nothing without knowing what was just discussed. You need a thin context layer that tracks conversation state, extracts entities (budget, name, phone number), and builds the prompt for each LLM call.

A mid-tier LLM with good conversation context will outperform a frontier model with no context. Build context management before you worry about model selection.

The design doesn't need to be complex. A simple class that tracks history, entities, and state, and builds a system prompt from them, handles the vast majority of real-world voice agent scenarios. Resist the urge to build a full NLU pipeline. The LLM's context window does most of that work for you.

Text-to-speech: queue, don't wait

TTS at high quality takes 500–1000ms for a full sentence. If you wait for the complete synthesis before playing audio, you're compounding every other latency. The pattern to use:

• Send the first sentence to TTS as the LLM generates it
• Queue subsequent sentences so they're ready as the first finishes playing
• Never wait for the full TTS response: stream audio chunks as they arrive

ElevenLabs Turbo v2.5 (500–1000ms, high quality) and Google Cloud TTS (1–2s, good quality) are the two most common choices. OpenAI's TTS sits in between at $0.015/1k characters. The difference between providers matters less than whether you're streaming the output or waiting for it.

Failure handling is not optional

Real-time systems fail. Networks drop. STT misrecognizes. LLM calls time out. The difference between a professional voice agent and a frustrating one is whether you've planned for these scenarios before they happen.

The key insight is that you have less than 100ms to decide what to do when something fails. That decision needs to be pre-planned, not reasoned about in the moment:

async function processInput(audio) {
  try {
    const text = await timeout(stt.transcribe(audio), 400);
    const reply = await timeout(llm.generate(context, text), 300);
    return await timeout(tts.synthesize(reply), 300);
  } catch (err) {
    if (err.component === 'stt') return tts.synthesize("Sorry, I didn't catch that.");
    if (err.component === 'llm') return tts.synthesize("Let me connect you with someone.");
    if (err.component === 'tts') return sendSMS(storedReply); // fallback channel
  }
}

Five lessons from production

1. Latency matters more than accuracy. Users judge how fast you respond, not how correct you were. Optimize for latency first. Accuracy improvements that cost latency are usually the wrong trade.
2. Stream everything. Streaming every component (STT interim results, LLM tokens, TTS audio chunks) reduces perceived latency by 200-500ms without changing actual backend performance.
3. Know your fallbacks before you need them. You have no time to think when a component fails. Map every failure mode to a scripted response before you ship anything to production.
4. Phone audio is a different medium. 8kHz, narrowband, compressed. Models trained on clean audio behave differently. Use phone-specific STT models and test with real carrier audio, not clean microphone input.
5. You can't fix what you can't measure. Track component-level latencies, STT accuracy, LLM error rate, and outcome metrics (booking rate, transfer rate) from day one. Aggregate latency alone won't tell you where to optimize.

Closing thoughts

Building production voice agents is a systems engineering problem, not an AI research problem. The LLM is genuinely the easy part now. The hard work is making five real-time services cooperate inside a one-second window, gracefully handling failure in every one of them, and maintaining enough observability to know when something is quietly degrading.

Start simple: Deepgram for STT, Claude Haiku or GPT-4o Mini for the LLM, ElevenLabs for TTS. Measure everything from the first call. Optimize latency before accuracy. Build fallbacks before you think you need them.

The goal is a voice agent that is fast, reliable, and forgettable, in the best possible way. Callers should hang up having accomplished what they called for, with no memory of anything going wrong.

If you're building voice agents at scale and want to skip the infrastructure work, Rymi provides a full voice stack via REST API: STT, LLM, TTS, barge-in, and post-call transcripts, all handled for you.