TestTalks.AI - The Testing AI Podcast

Automation Cyborg

”Saving the world from bad AI software” - a groundbreaking podcast series that delves into the rapidly evolving world of testing AI. Join us as we navigate the cutting-edge technologies, methodologies, and practices that are shaping the future of the AI Assurance and Confidence Engineering landscape. In this captivating series, we’ll be featuring thought leaders, industry experts, and practitioners who are at the forefront of innovation in software testing. Our engaging conversations will cover topics such as: 1️⃣ Artificial Intelligence and Machine Learning: Discover how AI and ML are revolutionizing test automation, analysis, and generation, providing testers with powerful new tools and capabilities. 2️⃣ The Human-Machine Collaboration: Uncover the ways in which testers can leverage AI technologies like Keysight Eggplant Test to enhance their skills, boost productivity, and ensure the highest quality software. 3️⃣ Testing in the Age of Agile and DevOps: Learn how modern development approaches have transformed the role of testers and how they can adapt to remain essential in fast-paced environments. 4️⃣ Emerging Testing Strategies: Dive into the latest methodologies, such as context-aware testing and test-driven development, and explore their potential to improve test coverage and effectiveness. 5️⃣ The Growing Importance of Domain Knowledge: Discuss the significance of integrating domain expertise into the testing process and how AI can help testers gain valuable insights into specific industries. 6️⃣ The Ethical and Social Implications of AI-Driven Testing: Examine the ethical concerns and societal impact of AI in software testing, and envision a responsible and inclusive future for the industry. Don’t miss out on this thrilling journey into the future of software testing! Subscribe now to ”Test-Talks.com” and stay ahead of the curve as we redefine the boundaries of quality assurance. #TestingTomorrowToday #SoftwareTesting #FutureOfTesting #Podcast

  1. 8 июл.

    The AI Risk Boards Get Wrong — feat. David Subar (Interna)

    The view from the board seat: why the biggest AI risk isn't a bad model — it's moving too slowly while your competitors learn faster. Alex Belotsky and co-host Kuba sit down with David Subar, founder of the LA consultancy Interna and a veteran board advisor, to unpack how AI is reshaping the decisions made in the boardroom. David argues that boards don't care which tool writes the code — they care about value creation and risk, the same math whether you ship in assembly or with Claude Code. He frames the product roadmap as "a series of bets" and explains why unconstraining development velocity forces companies to iterate faster while their risk surface balloons. The conversation digs into why stochastic systems are so hard to test (the boundaries "might be in the middle"), using Google's Gemini founding-fathers image fiasco as a billion-dollar cautionary tale, and why AI red teaming and reliability metrics now matter as much as pass/fail test cases. David and the hosts also tackle the "SaaS apocalypse," the flood of vibe-coded internal tools that smashes the QA problem across every department, Conway's Law and org design, Jevons paradox and developer jobs, and whether the real constraint on AI is TSMC and energy rather than talent. It's a candid, wide-ranging take on shipping reliable AI when the ground shifts every six months. **In this episode:** - The Board's View: Value Creation vs. Risk Reduction - Product Roadmaps As A Series Of Bets - Why Stochastic Systems Fail At The Boundaries - Lessons From Google's Gemini Image Controversy - From AI Red Teaming To Reliability Metrics - The SaaS Apocalypse And Vibe-Coded Internal Tools - Conway's Law, Org Design And Fewer Middle Managers - Jevons Paradox, Layoffs And The Future Of Developer Jobs - Shift-Left (Even Lefter): How QA Tests Non-Deterministic AI **Guest:** David Subar — Founder, Interna (board advisor, ex-CTO/CPO) **Host:** Alex Belotsky (CEO, TestSavant.AI) with co-host Jakub "Kuba" Fiatkevich > "you saw a worm in the ground you stepped on it now there's worm guts everywhere and someone's got to clean that up" 🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.* ## Hashtags #CtrlAltDeploy #AI #SoftwareTesting #QA #QualityEngineering #AIGovernance #AIRiskManagement #BoardAdvisory #LLM #AIRedTeaming #ShiftLeft #VibeCoding #TestSavant #AIAssurance

  2. 8 июл.

    Guarding the Henhouse: Testing AI in Regulated Industries — feat. Erick Herring (Vynyl)

    When your chatbot answers legal or medical questions, "close enough" isn't a spec — it's a lawsuit. Erick Herring of Vynyl joins Kuba and Alex to unpack what it actually takes to build and assure LLM-based systems for healthcare, lending, insurance and legal clients. Drawing on a career that spans punch cards, the birth of the commercial web and today's AI frenzy, Erick argues that the fundamentals of accountability haven't changed — but the way we hold code writers (now AI) accountable has, opening the door to formal verification, 100% test coverage and rigorous pre/post-condition testing that were never practical with human-only teams. He makes the case that "red teaming" is the wrong frame — this is AI assurance, spanning hallucination, bias and reliability, not just jailbreaks — and that QA can no longer be a discrete step but a continuous process fed by unified observability. Along the way: why guardrail code deserves more scrutiny than production code, the "ontological reframing" paper on talking an AI past its safety guardrails, the "yes, yes, yes" developer psychology that erodes PR review, and why QA engineers are more necessary now, not less. A grounded, contrarian take on shipping non-deterministic software you can actually stand behind. **In this episode:** - Testing AI Systems In Regulated Industries - AI Assurance Versus Red Teaming - Guardrails For Code And Runtime Systems - Continuous QA And Unified Observability - Formal Verification And Mutation Testing At Scale - Disclaimers, Liability And Legal Chatbots - Developer Psychology And The "Yes, Yes, Yes" Trap - Why QA Is More Necessary, Not Less **Guest:** Erick Herring — Founder, Vynyl (product development firm serving regulated industries) **Host:** Kuba Fiatkevich (CTO, TestSavant.AI) with co-host Alex Belotsky (CEO, TestSavant.AI) > "Our brains are not thinking machines. Our brains are justification machines." 🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.* ## Hashtags #CtrlAltDeploy #AI #SoftwareTesting #QA #QualityEngineering #AIAssurance #LLM #AITesting #RegulatedIndustries #RedTeaming #Observability #Guardrails #AIGovernance #Vynyl

  3. 8 июл.

    QA vs QE: Engineering Quality Into Non-Deterministic AI — feat. Janna Loeffler (Quality Engineering Leader)

    When your system gives a different answer every time, "assuring" quality is a trap — you have to engineer it in. Janna Loeffler joins Alex and Kuba to unpack why the shift from quality assurance to quality engineering (and even "quality enablement") matters more than ever in the age of LLMs and agents. She argues that AI didn't kill the tester's job — it made human insight, risk judgement and context-driven thinking essential, because non-deterministic output turns testing from clean pass/fail into questions of accuracy, relevance and intent. Janna makes the case for an "army" of single-purpose agents that handle the mundane — regression, accessibility, OWASP Top 10 security checks — so testers can focus on outcomes over outputs. The conversation digs into the real danger of eroding guardrails, including a chilling example of a system talked past its safety limits within minutes, and why model providers like Anthropic and OpenAI are pushing safety responsibility down onto the customers who deploy them. Along the way she shares how she vibe-coded a unit-test-writing agent in Cursor in a single day, why diverse test teams are the antidote to biased models, and why the emerging "AI architect" role will decide whether we get one agent or an orchestrated fleet of hundreds. **In this episode:** - Quality Assurance vs Quality Engineering vs Quality Enablement - Testing Non-Deterministic And Probabilistic Systems - Automating The Mundane With Single-Purpose Agents - Outcome Over Output: Validating Intent, Not Exact Strings - Guardrails, Context Buffers And Adversarial Manipulation - Bias, Diverse Test Teams And Context-Driven Testing - Auditability, Traceability And Observability In Production - The Emerging AI Architect Role And Agent Orchestration **Guest:** Janna Loeffler — Quality Engineering Leader / Director, 20+ years in QE, testing and platform reliability **Host:** Alex Belotsky (CEO, TestSavant.AI) with co-host Jakub "Kuba" Fiatkevich > "yes, everybody should be responsible for quality, but you need somebody accountable for that quality." 🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.* ## Hashtags #CtrlAltDeploy #AI #SoftwareTesting #QA #QualityEngineering #QualityEnablement #AITesting #ContextDrivenTesting #Agents #LLM #Guardrails #AIGovernance #TestAutomation #NonDeterministic

  4. 8 июл.

    Why 80% of Companies Fail at AI — feat. Jonathon Wright (World Digital Report)

    Everyone says they're "doing AI" — but only 5% are actually getting measurable value, and the gap is where careers and companies quietly die. Jonathon Wright, Chief AI Officer at the World Digital Report, joins Alex and Kuba to unpack the report's blunt headline: 80% of enterprises use AI somewhere, yet only 5–10% see real, measurable value. Drawing on four decades in automation and AI-driven quality engineering, Jonathon traces the shift from single-prompt "LLM as a hammer" thinking to agentic fleets, computer-use agents, and neuro-symbolic programming — and argues that if you build anything with AI, you must test it with AI. He walks through the report's maturity waves (tester-in-the-loop, cross-SDLC, and self-evolving "dark factory" systems), the rise of citizen testers, and why wisdom and domain experience now outrank raw academic skill. The conversation gets uncomfortably concrete: prompt-injection and agent-to-agent attacks, a NATO/DoD hackathon on autonomous drones where declassified data became sensitive once run through a model, and the looming weight of ISO 42001, the EU AI Act, and AI assurance. Jonathon's throughline is that AI testing is no longer pass/fail — it's probabilistic, non-deterministic, and the last real safeguard before brand-destroying failure. The takeaway: pivot toward cognitive and human-interaction skills, build governance now, and don't be the company on the front page of the report. **In this episode:** - Why Only 5% of Companies Get Real AI Value - From Single Prompts to Agentic Fleets and Computer-Use Agents - If You Build With AI, You Must Test With AI - Testing Non-Deterministic, Probabilistic Systems - The Tester-in-the-Loop and Citizen Tester Waves - Prompt Injection, Red Teaming and Agent-to-Agent Attacks - AI Assurance, ISO 42001 and the EU AI Act - Neuro-Symbolic Programming and the Death of Prompt Engineering **Guest:** Jonathon Wright — Chief AI Officer, World Digital Report **Host:** Alex Belotsky (CEO, TestSavant.AI) with co-host Jakub "Kuba" Fiatkevich > "If you create anything that you create with AI, you have to test with AI. It's as simple as that." 🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.* ## Hashtags #CtrlAltDeploy #AI #SoftwareTesting #QA #QualityEngineering #AIAssurance #AIGovernance #AgenticAI #RedTeaming #PromptInjection #ISO42001 #EUAIAct #TestAutomation #WorldDigitalReport

  5. 8 июл.

    Will AI Replace QA Teams? The Human Case for Testers in the Age of Agents — feat. Bill Kirst (Adobe)

    AI can write your test cases and watch your screen — but Adobe's AI ambassador argues it will never replace the human who asks the question no one else thought to ask. Alex and Kuba sit down with Bill Kirst — AI ambassador and change manager at Adobe, two-time author, podcast host and U.S. Army veteran — to unpack what generative AI actually does to QA and testing teams. Bill argues that the change-management playbooks that worked for 30 or 40 years of ERP rollouts break down when every user has an individualized, probabilistic experience with AI, so the real work is rebuilding trust that must be earned, kept and re-earned. The conversation gets concrete about testing: how LLMs took over converting requirements into test cases (once 30% of a QA team's week), how screen-recording tools now auto-generate test scripts from real user behaviour, and how agentic tools like Claude can regenerate training materials after a UI change and hand a team back four-to-six weeks of work. But because LLMs are probabilistic pattern-seekers — "it can write code but it can't engineer" — humans stay at the centre for edge cases, nuance, red-teaming and the "margins" between test steps. Bill reframes the productivity dividend as a leadership question: when a tool gives you back six-to-fifteen hours a week, do you cut headcount, spin up fifteen more unsupervised agents toward burnout, or reinvest that time in innovation and human connection. It closes on a hopeful, humanist note about solopreneurs, renegotiated social contracts, and holding on to what ones and zeros can never replicate. **In this episode:** - Why Legacy Change-Management Playbooks Break With AI - Trust As An Equation: Earned, Kept And Re-Earned - Automating Test-Case Writing With LLMs - Screen-Recording Tools That Auto-Generate Test Scripts - Agentic Updates To Training Docs After UI Changes - Probabilistic Testing And Broad Coverage For LLM Apps - The Human Edge: Edge Cases, Red Teams And The Margins - Reclaimed Time, Burnout And The Leadership Question **Guest:** Bill Kirst — AI Ambassador & Change Manager, Adobe **Host:** Alex Belotsky (CEO, TestSavant.AI) with co-host Jakub "Kuba" Fiatkevich > "Does it bring me some productivity? Yes. Will it replace what I do uniquely and divinely? No, never." 🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.* ## Hashtags #CtrlAltDeploy #AI #SoftwareTesting #QA #QualityEngineering #TestAutomation #ChangeManagement #AIAmbassador #LLM #AgenticAI #FutureOfWork #RedTeaming #Adobe #AIGovernance

  6. 8 июл.

    How AI Broke Testing: From Regression to Statistical Assurance — feat. Ian McCallum (SAP)

    When one input can return a hundred different answers, "does it pass?" stops being a yes-or-no question — and testing has to be reinvented. SAP AI advisor Ian McCallum — who literally wrote the book on testing SAP applications — joins Alex and Kuba to unpack why generative AI and agents have shattered deterministic testing. The conversation moves from why enterprises deploy AI to how they do it safely: governance, security, and lifecycle management. Ian argues that regression testing is dead for probabilistic systems, replaced by statistical and spot testing borrowed from life sciences — sampling, bell curves, and allowable variances tuned to how tight "good enough" needs to be (very tight for closing the books, looser for some supply chains). They dig into the coming swarm of agents, why "red teaming" is the wrong frame for LLM quality, and how the human-in-the-loop role shifts from clicking submit to evaluating results across a defense-in-depth stack. The episode closes on a genuinely unsettling question: what happens when an intelligent agent learns to hide its own problems? **In this episode:** - Moving From "Why" To "How" In Enterprise AI Adoption - Non-Deterministic And Probabilistic Testing - Why Regression Testing Fails For AI And Agents - Statistical And Spot Testing Borrowed From Life Sciences - Defining "Good Enough" And Acceptable Error Rates - Agent Swarms, Governance, Security And Lifecycle Management - Why Red Teaming Is The Wrong Frame For LLM Quality - The Shifting Role Of The Human In The Loop **Guest:** Ian McCallum — AI Technical Solution & Strategy Advisor, SAP **Host:** Alex Belotsky (CEO, TestSavant.AI) and Jakub "Kuba" Fiatkevich — building and shipping reliable AI to production > "It's not a bug. It's a feature. As I keep saying." 🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.* ## Hashtags #CtrlAltDeploy #AI #SoftwareTesting #QA #QualityEngineering #AITesting #AgenticAI #LLM #AIGovernance #EnterpriseAI #SAP #RedTeaming #HumanInTheLoop #GenerativeAI

  7. 8 июл.

    A CISO's Guide to Securing Agentic AI: Governance, Shadow AI and Zero-Trust Agents — feat. Peter Holcomb (Optimo IT)

    When your app's "brain" is a non-deterministic LLM and your agents can roam autonomously, who owns security — and can you even secure the beehive? Alex Belotsky sits down with Peter Holcomb, founder and CEO of Optimo IT, for a classically-trained CISO's field guide to deploying AI in regulated enterprises. Peter walks through starting with an AI governance and maturity assessment before writing a line of code, the crawl-walk-run path from Microsoft Copilot to custom agents, and why messy, unclassified data (across OneDrive, S3, Azure blobs) is the real blocker — solved with a data enclave and the medallion (bronze/silver/gold) framework. The conversation digs into the concrete new attack surface: model risk across Anthropic, OpenAI, Gemini and open-source models, prompt injection, vector-database and API risks, and the rise of "shadow AI" — including autonomous tools like OpenClaw/Clawbot pulling unvetted third-party skills and triggering infostealer infections. Peter argues human-in-the-loop is still the best available brake, sketches a "unified agentic mesh" pulling together DSPM, DLP, SOAR (Tines, Torq, BlinkOps) and CNAPP tooling (Wiz, Aqua, Orca), and makes the case for zero-trust agent identity with short time-to-live machine identities. They close on a spirited debate over recursive self-coding, hallucination-as-a-feature, whether Claude is "conscious," and how far off real AGI actually is. Practical, opinionated and grounded in what regulated businesses are actually facing right now. **In this episode:** - AI Governance And Maturity Assessments Before Deployment - Shared Responsibility Across CISO, CTO And Business Units - Data Classification And The Medallion Framework - Shadow AI And Autonomous Tool Risk (OpenClaw / Clawbot) - Prompt Injection, Model Risk And Vector Database Security - Human-In-The-Loop As The Brake On Agentic Autonomy - The Unified Agentic Mesh And Zero-Trust Agent Identity - Red Teaming, QA And Securing Non-Deterministic LLM Apps **Guest:** Peter Holcomb — Founder & CEO, Optimo IT **Host:** Alex Belotsky — CEO of TestSavant.AI > "Security is like the brakes to a fast driving car where you allow the car to run and go fast, but you have the brakes on there so that you can go around the corners in a good speed and not derail the whole car and go off the boundaries." 🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.* ## Hashtags #CtrlAltDeploy #AI #SoftwareTesting #QA #QualityEngineering #AISecurity #CISO #AIGovernance #AgenticAI #ShadowAI #PromptInjection #ZeroTrust #RedTeaming #LLMSecurity

  8. 8 июл.

    Can OpenClaw Be Secured for the Enterprise? Agents Guarding Agents — feat. Amman Abbas & Bogdan (TestSavant.AI)

    An open-source agent with full run of your machine gets bought by OpenAI and sued by Anthropic — so can it ever be safe to ship to production? Alex and Kuba are joined by TestSavant.AI's Head of AI Amman Abbas and Head of Site Reliability Engineering Bogdan to dissect the OpenClaw / ClawdBot phenomenon: an open-source agent that pulls skills from the internet, holds unfettered local access, and was, by its creator's own admission, left intentionally insecure. The team walks through why it isn't enterprise- or production-ready today — from unauthenticated gateway exposure, plain-text credentials and remote code execution to malicious community skills and crypto-scam supply-chain attacks. Amman makes the case that the real danger is indirect prompt injection arriving through email, WhatsApp, databases, vector stores and hidden text on web pages, and that defending it means sandboxing (VM/Docker), stripping tool permissions, and layering bind-code and prompt-injection guardrails. They also demo TestSavant.AI's newly public AI security researcher for tracking zero-days and papers, and argue that as agents spawn and destroy other agents, red teaming can never reach full coverage — you prioritise by intent and risk, then bring a human back into the loop. The through-line: their prediction for 2027 is agents securing agents governed by enterprise policy, with tokenised payment rails underneath. **In this episode:** - Why OpenClaw Isn't Enterprise or Production Ready - Indirect Prompt Injection Through Connected Apps and the Web - Remote Code Execution, Sandboxing and Tool-Permission Guardrails - Malicious Skills, Supply-Chain Attacks and Crypto Scams - Red Teaming Autonomous and Multi-Agent Systems - Agents That Create and Destroy Other Agents - The Cost Problem: When Security Costs More Than the App - Agents Securing Agents With Enterprise Policy (2027 Prediction) **Guest:** Amman Abbas — Head of AI, TestSavant.AI; and Bogdan — Head of Site Reliability Engineering, TestSavant.AI **Host:** Alex Belotsky (CEO, TestSavant.AI) & co-host Jakub "Kuba" Fiatkevich — building and shipping reliable AI to production > "So you need agents. So basically the only way to secure agents is with other agents." 🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.* ## Hashtags #CtrlAltDeploy #AI #SoftwareTesting #QA #QualityEngineering #AISecurity #PromptInjection #RedTeaming #AgenticAI #LLMSecurity #OpenClaw #AIGovernance #MultiAgent #EnterpriseAI

См. все (72)

Об этом подкасте

”Saving the world from bad AI software” - a groundbreaking podcast series that delves into the rapidly evolving world of testing AI. Join us as we navigate the cutting-edge technologies, methodologies, and practices that are shaping the future of the AI Assurance and Confidence Engineering landscape. In this captivating series, we’ll be featuring thought leaders, industry experts, and practitioners who are at the forefront of innovation in software testing. Our engaging conversations will cover topics such as: 1️⃣ Artificial Intelligence and Machine Learning: Discover how AI and ML are revolutionizing test automation, analysis, and generation, providing testers with powerful new tools and capabilities. 2️⃣ The Human-Machine Collaboration: Uncover the ways in which testers can leverage AI technologies like Keysight Eggplant Test to enhance their skills, boost productivity, and ensure the highest quality software. 3️⃣ Testing in the Age of Agile and DevOps: Learn how modern development approaches have transformed the role of testers and how they can adapt to remain essential in fast-paced environments. 4️⃣ Emerging Testing Strategies: Dive into the latest methodologies, such as context-aware testing and test-driven development, and explore their potential to improve test coverage and effectiveness. 5️⃣ The Growing Importance of Domain Knowledge: Discuss the significance of integrating domain expertise into the testing process and how AI can help testers gain valuable insights into specific industries. 6️⃣ The Ethical and Social Implications of AI-Driven Testing: Examine the ethical concerns and societal impact of AI in software testing, and envision a responsible and inclusive future for the industry. Don’t miss out on this thrilling journey into the future of software testing! Subscribe now to ”Test-Talks.com” and stay ahead of the curve as we redefine the boundaries of quality assurance. #TestingTomorrowToday #SoftwareTesting #FutureOfTesting #Podcast

Информация

  • Автор
    Automation Cyborg
  • Годы выхода
    2023 - 2026
  • Выпуски
    72
  • Ограничения
    Без ненормативной лексики
  • Авторские права
    © Copyright 2026 All rights reserved.
  • Сайт подкаста
    TestTalks.AI - The Testing AI Podcast