The Hardest Problem in AI: Evaluation in 2025 with Ian Cairns

In this episode of the ODSC AI Podcast, host Sheamus McGovern speaks with Ian Cairns, cofounder and CEO of Freeplay, a platform built to help teams evaluate, monitor, and iterate on LLM and agent-based systems in production. Ian brings a deep product background from Twitter, Gnip, and Mapbox, and offers an insider’s look into what it actually takes to make AI work beyond the prototype phase. The conversation centers on evaluation — widely regarded as one of the most difficult and underdeveloped aspects of deploying AI in 2025. Key Topics Covered: - The real-world AI maturity curve: from vibe prompting to production - Offline vs. online evaluation: definitions, trade-offs, and tooling - Why teams struggle post-deployment — and how to break through the “we don’t know what’s going wrong” phase - Evaluation challenges with agents, memory, RAG, and tool use - The role of observability, telemetry, and human-in-the-loop review - Lessons learned from Freeplay customers, including Postscript - The growing importance of domain experts in evaluation workflows - Building multi-layer eval architectures for agent systems - Voice agent challenges — like turn detection and latency - Emerging roles like AI Evaluation Engineer and how orgs should staff for evaluation maturity Memorable Outtakes: - "The most mature teams start with their evals. They define what good looks like, then hill-climb toward that metric." - "The breakthrough in quality comes from people getting close to the data. Sometimes, thousands of rows." References & Resources: - Freeplay website: https://www.freeplay.ai - Deployed: The AI Product Podcast by Freeplay: https://open.spotify.com/show/6nZS3a7iYb2EzHcl78iNmi?si=de766e786a41461c&nd=1&dlsi=0cb3351f79644bfc - Freeplay blog: https://www.freeplay.ai/blog - Freeplay community newsletter: https://freeplay.ai/newsletter - PipeCat (open-source voice agent toolkit): https://github.com/pipecat-ai/pipecat - OpenTelemetry (agent observability framework): https://opentelemetry.io/ - Postscript (Freeplay customer case mentioned): https://www.postscript.io - Colorado AI community meetups: https://www.boulderaibuilders.org/ Speaker Bio: Ian Cairns is the CEO and co-founder of Freeplay. Previously, he served as Head of Product for Twitter’s Developer Platform, where he helped grow their enterprise data business from $40M to $400M ARR. He’s also worked at Gnip (acquired by Twitter), Mapbox, and in the Obama administration on open data initiatives. LinkedIn: https://www.linkedin.com/in/iancairns/ Sponsored by: 🔥 ODSC West 2025 – The Leading AI Training Conference Join us in San Francisco from October 28th–30th for expert-led sessions on generative AI, LLMOps, and AI-driven automation. Use the code podcast for 10% off any ticket. Learn more: https://odsc.com/california