Voice AI applications are revolutionizing how we interact with technology, but latency remains the biggest barrier to creating truly conversational experiences. When users have to wait seconds for a response, the magic of natural conversation is lost. In this comprehensive guide, we'll show you how to build a voice agent in Vapi that achieves an impressive ~465ms end-to-end latency—fast enough to feel truly conversational. Vapi Understanding the latency challenge Before diving into the configuration, it's crucial to understand that voice agent latency comes from multiple components in the pipeline: Speech-to-Text (STT): Converting audio to text
Large Language Model (LLM): Processing and generating responses
Text-to-Speech (TTS): Converting text back to audio
Turn Detection: Determining when the user has finished speaking
Network Overhead: Data transmission delays Speech-to-Text (STT): Converting audio to text Speech-to-Text (STT) Large Language Model (LLM): Processing and generating responses Large Language Model (LLM) Text-to-Speech (TTS): Converting text back to audio Text-to-Speech (TTS) Turn Detection: Determining when the user has finished speaking Turn Detection Network Overhead: Data transmission delays Network Overhead The key to ultra-low latency is optimizing each component and minimizing unnecessary delays. The optimal configuration stack Our target configuration achieves the following breakdown: STT: 90ms (AssemblyAI Universal-Streaming)
LLM: 200ms (Groq Llama 4 Maverick 17B)
TTS: 75ms (Eleven Labs Flash v2.5)
Pipeline Total: 365ms
Network Overhead: 100ms (Web) / 600ms+ (Telephony)
Final Latency: ~465ms (Web) / ~965ms+ (Telephony) STT: 90ms (AssemblyAI Universal-Streaming) STT LLM: 200ms (Groq Llama 4 Maverick 17B) LLM TTS: 75ms (Eleven Labs Flash v2.5) TTS Pipeline Total: 365ms Pipeline Total Network Overhead: 100ms (Web) / 600ms+ (Telephony) Network Overhead Final Latency: ~465ms (Web) / ~965ms+ (Telephony) Final Latency Step 1: Configure speech-to-text with AssemblyAI AssemblyAI's Universal-Streaming API is currently one of the fastest STT options available, delivering transcripts in just 90ms. Universal-Streaming API Key Configuration Settings: Critical optimization: Disable formatting This is perhaps the most important STT optimization that many developers overlook. By setting Format Turns : false, you eliminate unnecessary processing time that adds latency. Modern LLMs are perfectly capable of understanding unformatted transcripts, and this single change can save precious milliseconds in your pipeline. Why this matters: Formatting processes like punctuation insertion, capitalization, and number formatting require additional computation. When every millisecond counts, these "nice-to-have" features become latency bottlenecks. Why this matters Step 2: Choose the right LLM - Groq's Llama 4 Maverick 17B The LLM is typically the highest latency component in your voice pipeline, making model selection critical. Groq's Llama 4 Maverick 17B 128e Instruct offers the perfect balance of speed and capability. Configuration: Why Groq + Llama 4 Maverick? Optimized Model: Llama 4 Maverick offers a best-in-class performance-to-cost ratio
Consistent Performance: 200ms processing time with minimal variance
Open Source: Cost-effective compared to proprietary alternatives Optimized Model: Llama 4 Maverick offers a best-in-class performance-to-cost ratio Optimized Model Consistent Performance: 200ms processing time with minimal variance Consistent Performance Open Source: Cost-effective compared to proprietary alternatives Open Source Pro Tip: Keep your maxTokens relatively low (150-200) for voice applications. Users expect concise responses in conversation, and shorter responses generate faster. Pro Tip Step 3: Implement lightning-fast TTS with Eleven Labs Flash v2.5 Eleven Labs Flash v2.5 is engineered specifically for low-latency applications, achieving an impressive 75ms time-to-first-byte. Configuration: Key Settings Explained: Optimize Streaming Latency: Set to 4 for maximum speed priority
Voice Selection: Choose simpler voices for faster processing
No Style Exaggeration: Higher values may increase latency slightly Optimize Streaming Latency: Set to 4 for maximum speed priority Optimize Streaming Latency Voice Selection: Choose simpler voices for faster processing Voice Selection No Style Exaggeration: Higher values may increase latency slightly No Style Exaggeration Step 4: Optimize turn detection settings This is where many developers unknowingly sabotage their latency optimization. Vapi's default turn detection settings include wait times that can add 1.5+ seconds to your response time—completely negating all your other optimizations. Critical configuration in advanced settings: Before: After: Why this matters as much as model choice: The default settings often include: Wait Seconds: 0.4s (unnecessary delay)
On PunctuationSeconds: 0.1s (unnecessary delay)
On No Punctuation Seconds: 1.5s (waiting when no punctuation detected)
On Number Seconds: 0.5s (unnecessary delay) Wait Seconds: 0.4s (unnecessary delay) Wait Seconds On PunctuationSeconds: 0.1s (unnecessary delay) On PunctuationSeconds On No Punctuation Seconds: 1.5s (waiting when no punctuation detected) On No Punctuation Seconds On Number Seconds: 0.5s (unnecessary delay) On Number Seconds Since our STT has formatting disabled, the system would default to the 1.5s "no punctuation" delay—adding 1500ms to a pipeline that we've optimized to 365ms (4x!). This single setting can make or break your latency goals. Network considerations and deployment Network considerations and deployment Web vs. telephony latency: Web (WebRTC): ~100ms network overhead
Telephony (Twilio/Vonage): 600ms+ network overhead Web (WebRTC): ~100ms network overhead Web (WebRTC) Telephony (Twilio/Vonage): 600ms+ network overhead Telephony (Twilio/Vonage) Deployment tips: Choose regions wisely: Deploy close to your users
Consider CDN: For global applications, use edge locations
Monitor performance: Set up latency monitoring and alerts
Test thoroughly: Network conditions vary significantly Choose regions wisely: Deploy close to your users Choose regions wisely Consider CDN: For global applications, use edge locations Consider CDN Monitor performance: Set up latency monitoring and alerts Monitor performance Test thoroughly: Network conditions vary significantly Test thoroughly Testing and monitoring your configuration Key metrics to track: End-to-end latency: Time from user stops speaking to agent starts responding
Component breakdown: Individual STT, LLM, TTS timings
Network overhead: Measure actual vs. expected network delays
User experience: Conduct user testing for perceived responsiveness End-to-end latency: Time from user stops speaking to agent starts responding End-to-end latency Component breakdown: Individual STT, LLM, TTS timings Component breakdown Network overhead: Measure actual vs. expected network delays Network overhead User experience: Conduct user testing for perceived responsiveness User experience Common pitfalls and troubleshooting 1. Forgetting turn detection settings Problem: Great model configuration, but 1.5s delays remain Problem Solution: Always check and optimize startSpeakingPlan settings Solution 2. Over-engineering prompts Problem: Long system prompts increase LLM processing time Problem Solution: Keep prompts concise and specific Solution 3. Ignoring network conditions Problem: Perfect configuration, but poor real-world performance Problem Solution: Test in various network conditions and locations Solution 4. Choosing quality over speed Problem: Using high-quality but slower models Problem Solution: For voice, prioritize speed; users value responsiveness over perfection Solution Conclusion Building a voice agent with ~465ms end-to-end latency is achievable with the right configuration and attention to detail. The key insights are: Every component matters: Optimize STT, LLM, and TTS individually
Turn detection is critical: Default settings can destroy your latency goals
Disable unnecessary features: Formatting and other "nice-to-haves" add latency
Test in realistic conditions: Network overhead varies significantly by deployment Every component matters: Optimize STT, LLM, and TTS individually Every component matters Turn detection is critical: Default settings can destroy your latency goals Turn detection is critical Disable unnecessary features: Formatting and other "nice-to-haves" add latency Disable unnecessary features Test in realistic conditions: Network overhead varies significantly by deployment Test in realistic conditions By following this configuration and understanding the principles behind each optimization, you'll create voice agents that feel truly conversational. Remember, in voice AI, perceived speed often matters more than absolute accuracy—users will forgive minor imperfections but won't tolerate slow responses. The future of voice AI lies in these ultra-responsive interactions. With this guide, you're now equipped to build voice agents that meet users' expectations for natural, real-time conversation.

Keep

AssemblyAI 

How to Build the Lowest Latency Voice Agent in Vapi: Achieving ~465ms End-to-end Latency

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

6 Best APIs for Topic Detection in 2022

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

6 Best APIs for Topic Detection in 2022

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps