Voice agents are one of the hottest use cases in enterprise right now, but also one of the hardest to actually take live. Getting latency low enough to feel human without dumbing down the responses, making reliable tool calls to a CRM without dropping the customer mid-call, building fallback models for when Anthropic or OpenAI are running hot. None of it is as simple as the demos make it look. Last week I hosted a fireside chat with five eng leaders who deal with this stuff every day: Basia Sudol (Head of Enterprise Solutions, Decagon), Varun Singh (CPTO, Daily), Steven Diaz (FDE Manager, Vapi), Tyler D'Silva (Founding FDE, Retell AI), and Sudarshan Kamath (Founder, Smallest AI). We get into why nobody serious is shipping real-time voice-to-voice yet, why LLMs forget the middle of your prompt (and what that does to your architecture), why a giant prompt quietly destroys your unit economics, and why voice agent costs are now being compared directly against human labor. Plus the stuff nobody warns you about: turn-taking, HIPAA constraints, why outbound is easier than inbound, why getting an exec to actually like the voice can be harder than any model problem, and more! Chapters below: 00:00 Intro 00:26 Meet the panel 01:23 Daily, WebRTC, and 20 years of building voice 03:49 How Smallest AI made real-time TTS work 05:23 Why Decagon moved into voice 08:16 How Vapi and Retell think about the stack 11:14 Forward deployed vs solutions engineering 16:48 Voice agent architecture, explained simply 22:11 Cascade vs speech-to-speech: the real tradeoff 28:11 Hybrid pipelines and mixing models 32:16 Accents, multilingual, and getting Singlish right 35:32 Prompts vs workflows, and the latency fight 44:26 How you actually evaluate a voice agent 49:04 Simulation-based evals 49:49 What production metrics really look like 51:53 Building a QA framework that scales 54:53 Evaluating speech-to-speech 58:10 Open source benchmarks 59:57 Why picking a voice is so subjective 01:02:03 Personalization and custom voices 01:03:20 Voice quality is solved, GPU efficiency is the new war 01:06:23 Why outbound calls work better than you'd think 01:10:38 Deploying in regulated industries (HIPAA, retention, audits) 01:12:43 Turn-taking, the hardest unsolved problem in voice 01:18:53 Where voice agents go in the next year 01:27:55 Audience Q&A: inside Smallest's Hydra model 01:32:15 The deployment problems nobody has solved yet 01:37:46 Closing thoughts and thanks