Prompt to Prod

Prompt to Prod
  • 0.0 (0)
  • Business

Prompt to Prod is a podcast about building with AI. Hosted by Aaron Mog, each episode features practitioners from security and engineering who are shipping real things with AI: the tools they build, the workflows that stuck, and the lessons learned along the way. Expect honest conversations about build vs. buy, guardrails, data and model choices, and what breaks on the way from a prompt to production. No hype and no vendor pitches, just people who build talking about what actually works.

Episodes

  1. 3d ago

    AI Hasn't Changed Incident Response | Prompt to Prod - with Kyle Smith

    AI hasn't really changed incident response. Same data, same regulations, same phishing emails. It just happens faster. Kyle Smith runs cyber incident response for a hospitality company after 16 years across retail, medical certification, industrial supply, higher education, healthcare and hospitality. We get into why he thinks AI moves the speed of IR but not its fundamentals, why automation should run up to a human decision point and stop, and how a marketing background changes the way a security team sells itself inside a company. Plus: the service recovery paradox as an IR strategy, using AI to sharpen skepticism instead of replacing your own writing, why chain of custody and legal privilege still need a human signature, whether you need AI to investigate AI, and why the frontier labs should build their own version of PCI. And if you run IR and want to start with AI this week: domain monitoring and takedowns. TIMESTAMPS 0:00 Cold open 0:34 Intro 0:42 Kyle's background: six industries, 16 years 3:07 IR hasn't changed: the data is the same 4:02 The economics of attacks just flipped 5:02 Patient, slow AI attackers 6:17 AI gives you the average. IR needs nuance 8:38 The service recovery paradox 11:29 Target, an HVAC vendor and chip cards 13:48 A marketing brain in security 14:17 Crack an egg into it 16:43 AI for budgets, decks and the AI flex 18:02 Using AI to enhance skepticism 21:48 AI as a garnish, and chain of custody 24:28 Automate to the decision point 28:02 Is IR getting ahead of itself? The cloud bill 30:31 Regulation has to catch up 32:50 YOLO security 34:16 Wins: realism over hype 37:14 Non-deterministic AI vs forensics; IR vs SOC 39:33 Old tactics made efficient: device code phishing 41:10 Do you need AI to investigate AI? 43:58 The worry meter and the credit card model 48:57 CrowdStrike and consumer perception 50:39 A PCI for the frontier labs 52:55 Where to start: domain monitoring and takedowns 56:50 A phishing-counseling agent 57:48 Where to find Kyle ABOUT KYLE Kyle Smith leads cyber incident response for a hospitality company. He has 16 years in IT infrastructure and cybersecurity, from help desk and server engineering to compliance and IR, and studied IT management and marketing. He recently spoke on leveraging AI in incident response at Kennesaw State. LinkedIn: https://www.linkedin.com/in/kyle-smith-9478a113 ABOUT PROMPT TO PROD Prompt to Prod is a show about building with AI — what actually works, what breaks, and what the best and worst of it looks like in practice. Host Aaron Mog talks to the engineers shipping real systems, not the people talking about them. #IncidentResponse #AISecurity #Cybersecurity #SOC #PCI #AIEngineering #Phishing

  2. Oct 20

    AI Leaked the API Key. Red Team Used It. | Prompt to Prod - with Michael Kouremetis

    A red team was ready for a week of reverse engineering. The model handed them the API key instead — it was sitting in the training data. Michael Kouremetis builds offensive cyber agents and evaluations on the Capabilities Team at Dreadnode, including DreadIndex, a living benchmark of how well models actually perform offensive security tasks. We get into why public benchmark scores didn't match what his team saw day to day, how models went from unreliable tool calling to "just give it a bash tool" in a year, and why his research has moved from "can it do this?" to scope and control. Plus: judge agents that sit beside the primary agent on every tool call, fast small models that escalate when unsure, the "golden trajectories" everyone wants for fine-tuning, why the harness matters less than it used to, and how to start building your first offensive agent this weekend. TIMESTAMPS 0:00 Cold open 0:28 Michael's background: MITRE, Caldera, Dreadnode 1:58 The Capabilities Team 3:53 What DreadIndex is and why they built it 5:39 What "better" means: cheaper and more reliable 8:52 From bad tool calling to "just give it bash" 1 1:21 So what? The shelf life of cyber evals 14:57 What customers see when the agents run 18:25 Do you need AI to stop AI? 21:39 YOLO security 23:21 The API key in the training data 25:46 Model asymmetry and special-access models 27:48 Sovereign stacks and on-prem models 2 9:13 Purpose-built models: capability in a box 34:54 Golden trajectories and fine-tuning 39:00 AI didn't create more criminals 41:01 Are we racing toward disaster? 47:21 Where to start: harness, task, DreadGOAD 53:01 Do harnesses still matter? 57:06 Trust, OpenRouter and on-prem 1:03:17 Canary tokens to follow the leak 1:05:45 Where to find Michael ABOUT MICHAEL Michael Kouremetis is on the Capabilities Team at Dreadnode, where he works on offensive cyber agents, evaluations for frontier labs, and DreadIndex. Before Dreadnode he spent about nine years at MITRE on cyber standards and R&D, including the Caldera adversary emulation platform. DreadIndex: https://dreadnode.io/research/dreadindex DreadGOAD (free cyber range): https://github.com/dreadnode/DreadGOAD ABOUT PROMPT TO PROD Prompt to Prod is a show about building with AI — what actually works, what breaks, and what the best and worst of it looks like in practice. Host Aaron Mog talks to the engineers shipping real systems, not the people talking about them. #AISecurity #AIAgents #OffensiveSecurity #RedTeam #LLM #AIEngineering #Cybersecurity

  3. Oct 14

    Treat Your LLM Like a Threat Actor | Prompt to Prod - with Assaf Kipnis

    Assaf Kipnis spent 13 years hunting adversaries at LinkedIn, Facebook, Google and ElevenLabs. Now he builds with AI the same way he hunted them — by assuming the model is a threat actor and writing code for everywhere it might go that you didn't ask for. Assaf and Aaron get into what it actually takes to build something you'd hand a client: why hallucinations are the wrong fear and agent decisions are the real one, why most of what he produces is code even though AI writes all of it, and the test he runs before reaching for a model at all — could I have done this 10 years ago? He walks through the "Kippy system" he built inside Claude Code to run his projects and his OSINT investigations, the document that blew up in his face when he let an LLM do it directly, and how he strips structure away from the model so it has less room to invent. Plus: why "we'll fight bad AI with good AI" is bananas, why a defender will never be faster than an attacker, why block lists never worked, and why he thinks billing by the hour is dead. TIMESTAMPS 0:00 Cold Open 0:36 Meet Assaf Kipnis 2:24 What He's Building: The Kippy System 6:47 The Investigation System That Runs OSINT for Him 8:17 Why Personal AI Assistants Still Don't Work 12:09 Hallucinations Aren't the Problem. Decisions Are. 14:59 False Positives and Why He Writes Code Instead 17:04 The Document That Blew Up in His Face 19:27 Taking Structure Away From the LLM 20:59 Using AI to Write the Thing That Does the Job 23:00 AI Is Infrastructure, Marketed to the Wrong People 26:30 Educating Clients Who Think It Just Works 29:45 Treating the LLM as a Threat Actor 31:28 "Automagically" Became "AI" 36:00 Could I Have Done This 10 Years Ago? 38:14 Consulting After AI: Just Give Them the Answer 42:25 Billing by the Hour Is Dead 45:12 What AI Hands the Threat Actors 46:06 The Soapbox: Block Lists Never Worked 49:29 A Defender Will Never Be Faster Than the Attacker 50:19 What to Do Instead: Understand Threat Actors 54:00 How Do You Prove Something That Didn't Happen? 57:42 We Can't Control Content Anymore. Go to Intent. 61:53 Pattern Recognition and the Gut-Feeling Problem 63:15 Where to Find Assaf ABOUT ASSAF Assaf Kipnis has spent 13 years across information security, trust and safety, and integrity, mostly doing social media and large-scale investigations. He was an investigator at LinkedIn, then spent about five years at Facebook where he managed and investigated on the e-crime team covering financially motivated actors. He did scaled investigations across Google's products and voice investigations at ElevenLabs. He now consults on AI, threat intelligence, fraud and security, helping companies build deterministic, trustworthy workflows. LinkedIn: https://www.linkedin.com/in/assafkipnis/ X: [CONFIRM HANDLE] Site: [CONFIRM URL] ABOUT PROMPT TO PROD Prompt to Prod is a show about building with AI — what actually works, what breaks, and what the best and worst of it looks like in practice. Host Aaron Mog talks to the engineers shipping real systems, not the people talking about them. #AISecurity #ThreatIntelligence #AgenticAI #AIEngineering #Cybersecurity #TrustAndSafety #BuildingWithAI

  4. Oct 7

    Don't Be a Meat Proxy for AI | Prompt to Prod - with Alex Beaver

    Alex Beaver's biggest problem with AI? It runs on GPUs. A security engineer who thinks about the SOC like a factory floor explains why the best AI projects end with less AI running in production, not more. Alex and Aaron get into what AI actually looks like inside a SOC: using an agentic SOC for fast coverage, then instrumenting it and converting its repeated tool calls back into cheap, deterministic workflows. Alex walks through his build process — writing the types and schemas himself, then letting Claude Code and Codex fill in the implementation — why models still get third-party schemas wrong, inventing a detection schema to do gap analysis, and why build vs. buy comes down to having someone to call at 3 AM. Plus: how worried defenders should be about offensive AI, where to start if you're new to this, and why "meat proxies" between Jira and an AI won't last. TIMESTAMPS 0:00 Cold open 0:35 Meet Alex Beaver 1:57 What AI Actually Looks Like in the SOC 3:04 Security Operations Through an Industrial Engineering Lens 4:23 Why Deterministic Matters (and the GPU Problem) 7:41 The Spectrum from People to Hot-Path Automation 10:33 Building a SOC from Scratch and the MSP Shake-Up 12:46 From Agentic SOC to Deterministic Workflows 15:18 Why Models Get Your Schemas Wrong 17:51 Write the Types First, Let AI Fill the Blanks 20:38 Rust, TypeScript and Typed Code for AI 22:55 AI for Threat Intel and Detection Engineering 26:20 Inventing a Detection Schema 29:38 Build vs. Buy: The 3 AM Phone Number 31:53 Real Wins: Shipping SOC Capabilities Faster 34:51 How Worried Should Defenders Be About Offensive AI? 39:03 Supply Chain Risk 40:53 What AI Is Bad At 43:16 Where to Start with AI 44:53 Cutting Through the Hype 47:18 Careers, Compression and Meat Proxies 50:28 Fundamentals Keep Coming Up ABOUT ALEX Alex Beaver is a security engineer at Paramount Skydance, focused mainly on SOAR and detection and response, plus internal tooling. Before that he built an automation platform at a mobile network, and he's long been interested in network orchestration and optimizing incident response. He ran education for one of the largest university cybersecurity clubs in the country right as ChatGPT arrived. He writes about AI in security on his blog, Secure at Scale: [LINK] ABOUT PROMPT TO PROD Prompt to Prod is a show about building with AI — what actually works, what breaks, and what the best and worst of it looks like in practice. Host Aaron Mog talks to the engineers shipping real systems, not the people talking about them. #AISecurity #SecurityOperations #AISOC #DetectionEngineering #AIEngineering #Cybersecurity #AgenticAI

  5. Sep 30

    The AI Security Control That Never Failed | Prompt to Prod - with Vince Ovando

    Vince Ovando thinks prompt injection isn't a real vulnerability — and he has millions of inference calls to back up why the industry's favourite defences aren't the ones that work. In this episode Vince breaks down harness engineering: why the model you pick matters far less than the code you wrap around it, why most problems should be solved with deterministic code before an LLM ever gets involved, and how grammar constrained decoding made every one of millions of test calls safe. We get into positive vs. negative security models, why he tests AI without running the model at all, agent permissions enforced at the token level, and knowledge graphs as a memory layer that lets a local model outperform a frontier one. TIMESTAMPS 0:00 Cold open 0:09 Introduction to AI Security and Vince's Background 3:10 Harness Engineering: The Key to AI Efficiency 6:09 The Role of Deterministic Code in AI Solutions 9:01 Contrarian Views on AI Security and Traditional Models 12:07 Positive Security Models vs. Traditional Security 15:08 AI Security vs. AI Safety: Understanding the Difference 18:13 The Importance of Not Trusting AI Models 21:02 Implementing Security in AI Architectures 24:02 Grammar Constrained Decoding: A New Approach to Security 27:14 User Permissions and AI: Building Secure Interactions 29:00 Harness Engineering in AI Security 31:04 Dynamic Tool Management for AI Agents 33:40 The Role of Grammars in AI Compliance 36:37 Building Custom AI Harnesses 39:53 Optimizing AI Performance with Knowledge Graphs 46:23 Designing Efficient Knowledge Graphs 49:22 Starting Out with AI: A Practical Playbook ABOUT VINCE Vince Ovando came up through security operations, moved into application security and development, and has spent the last two years deep in AI security. His work covers agentic security, red teaming, and code analysis. He built Tantalus (tantalus.io), a free two-round indirect prompt injection arena, and used it to run the millions of inference calls behind his white paper on grammar constrained decoding. ABOUT PROMPT TO PROD Prompt to Prod is a show about building with AI — what actually works, what breaks, and what the best and worst of it looks like in practice. Host Aaron Mog talks to the engineers shipping real systems, not the people talking about them. #AISecurity #PromptInjection #LLM #AIEngineering #Cybersecurity #AgenticAI #KnowledgeGraphs

  6. Sep 22

    AI Used 17,600 Events. APTs Use 50. | Prompt to Prod - with Justin Tiplitsky

    Prompt to Prod: an AI agent completed its goal in 17,600 events. The average advanced attacker, by Justin Tiplitsky's count, uses about 50. Justin, director of red team at Adobe, joins Aaron to talk about where AI actually helps a red team, where it's still loud and sloppy, and why handmade tradecraft is what will make AI useful on offense. We cover what changed when models started keeping context, why Justin doesn't let AI touch his code directly, how AI lowers the bar for zero days, using AI on huge recon datasets, turning red team output into detections defenders can use, the difference between AI red teaming and red teaming with AI, why you can give AI knowledge but not wisdom, and his advice for the next generation of red teamers. TIMESTAMPS 0:00 Cold Open 0:33 Welcome and meet Justin 2:01 How a red team leader feels about AI right now 3:43 What changed: models that keep context 6:13 Don't AI yourself into a corner: handmade tradecraft 8:52 AI lowers the bar for zero days 14:06 When to use AI on a red team 16:30 Recon and large datasets 19:19 Turning red team output into better detections 22:19 Are AI red team products any good? 25:05 AI red teaming vs. red teaming with AI 30:41 What Justin wants to build next 34:51 Knowledge vs. wisdom 38:41 Advice for the next generation of red teamers 42:00 Where to find Justin ABOUT JUSTIN Justin Tiplitsky is director of red team at Adobe, where he built the company's current red team. He previously worked on red teams at Microsoft and Intuit, and writes about strategic red teaming at redteambrief.com. ABOUT PROMPT TO PROD Prompt to Prod is a podcast about building with AI, hosted by Aaron Mog. Each episode, practitioners from security and engineering share what they're actually building, what breaks on the way to production, and what they'd do differently. #PromptToProd #RedTeam #AISecurity #OffensiveSecurity #Cybersecurity #BuildingWithAI #AIAgents

  7. Sep 21

    He Hasn't Hand-Written Code in 18 Months | Prompt to Prod - with Wyatt Roersma

    Prompt to Prod: Wyatt Roersma hasn't hand-written code in over 18 months. His day is code review: AI builds in a sandbox, tests its own work, and attaches a GIF of the finished UI to every pull request. Wyatt and Aaron get into how a longtime security practitioner builds with AI now, and where it still falls short. We cover build vs. buy for security teams, why Wyatt thinks everyone should build with AI, how he turns infrastructure into code so AI stops guessing, running Codex and Claude side by side with parallel agents, MCP, fine-tuning small models for YARA and MITRE ATT&CK mapping, why private data and subsidized tokens push toward private models, where models fall short on intent and access, and what a blue-team model should look like. TIMESTAMPS 0:00 Cold Open 0:45 Welcome and meet Wyatt 1:17 Wyatt's background: memory forensics to Rust 3:05 What he's building: a security coverage platform 8:06 How AI fits in: no hand-written code in 18 months 9:24 PRs that come with videos 12:59 Build vs. buy for security teams 14:24 Why everybody should build with AI 19:37 Where AI struggles: infrastructure 21:04 Wyatt's stack: Codex, Claude, and parallel agents 24:19 Infrastructure as code so AI stops guessing 29:05 MCP and threat intelligence services 30:44 Private models and fine-tuning small models 37:45 Private data, subsidized tokens, and the price cliff 39:40 Where models fall short: intent and access 46:37 What Wyatt wants to build next: a blue-team model 51:38 Model vs. harness: what security AI is missing 55:43 Where to find Wyatt ABOUT WYATT Wyatt Roersma got his start in memory forensics, contributing to Volatility around 2012. He worked at Target after the breach, building a custom Cuckoo sandbox for the threat detection operations team, later worked as a cyber range developer, and has run the GrrCON digital forensics and incident response challenge since 2013. He now works at Countermeasure Security, building a coverage platform, and publishes open-source projects, many in Rust, under Threatflux. Catch Wyatt's DFIR challenge at GrrCON in Grand Rapids, September 24–25. ABOUT PROMPT TO PROD Prompt to Prod is a podcast about building with AI, hosted by Aaron Mog. Each episode, practitioners from security and engineering share what they're actually building, what breaks on the way to production, and what they'd do differently. #PromptToProd #AISecurity #BuildingWithAI #DetectionEngineering #Rust #ClaudeCode #AIAgents

  8. Sep 21

    AI Said Everything Was on Fire | Prompt to Prod - with Pablo Vidal

    Prompt to Prod: when Pablo's team first pointed an AI analyst at their Okta logs, it treated a Monday holiday like an incident. Everything was on fire. Pablo and Aaron get into what it really takes to build with AI as a security engineer: giving models context, making sure your team can own what you build, and putting humans only where the risk demands it. We cover why everyone can build now and what guardrails that requires, which data you should never send to a model, build vs. buy and why you shouldn't build in a cave, what to do when AI generates 5,000 PRs a day, risk-classifying PRs so humans review only what matters, alert fatigue and the holiday-calendar fix, baselining and timelines in detection, cost and cheaper models as token prices catch up, why he isn't more worried about the AI threat wave, and why AI is still bad at writing. TIMESTAMPS 0:00 Cold Open 0:42 Welcome and meet Pablo 1:11 Pablo's background: from Cisco certs to AI security 2:35 How much of his day is AI 4:31 Enablement: everyone can build now 7:54 Guardrails, not gatekeeping 10:03 Sending sensitive data to models 15:17 Build vs. buy: don't build in a cave 19:12 Velocity vs. quality vs. security 22:57 Guardrails agents can read: risk-classifying PRs 26:46 Alert fatigue: how much can AI handle 28:20 Give AI context: the holiday-calendar false alarms 30:59 Baselining, timelines, and non-deterministic AI 36:05 Cost, local models, and deterministic scripts 41:02 The AI threat wave 45:22 What AI is bad at: writing 49:23 What Pablo is building next 50:59 Community and where to find Pablo ABOUT PABLO Pablo has spent about 11 years in security. He started with Cisco certifications, moved into security generalist work and cloud security on AWS, worked on cloud security and security operations at Rippling, then spent a year at Grow Therapy in an individual-contributor role where he went deep on AI. He recently joined Life360 as a senior staff engineer focused on AI security. ABOUT PROMPT TO PROD Prompt to Prod is a podcast about building with AI, hosted by Aaron Mog. Each episode, practitioners from security and engineering share what they're actually building, what breaks on the way to production, and what they'd do differently. #PromptToProd #AISecurity #BuildingWithAI #SecurityEngineering #DetectionEngineering #ClaudeCode #AIAgents

  9. Sep 21

    One AI Fix Found Four More Issues | Prompt to Prod - with Drew Pilarski

    One fix, four new issues. Security engineer Drew Pilarski went from nervous about AI, to excited, to a little tired, and he's honest about all three. He and Aaron get into what it's really like to build with AI every day: what it makes possible, the tooling you now have to keep alive, and why one AI fix can turn into four new issues. We cover build vs. buy when you're not a traditional developer, the boring corporate work AI does best, a LangGraph port that spiraled into "endless work," Aaron's loop of building with one model and validating with another, running parallel agents, frontier vs. private models as token costs catch up, where AI is already paying off in detection engineering, and what happens when attackers have models with no guardrails. TIMESTAMPS 0:00 Cold Open 0:34 Welcome and meet Drew 2:41 From nervous to excited to tired 7:00 Build vs. buy: keeping the tooling alive 7:43 The boring work AI does best 10:03 Recall Radar and the joy of building 13:18 Where AI let him down: LangGraph and the Ouroboros loop 15:22 Fable builds, Astra validates 18:22 Claude Code, Herdr, and the 10x engineer question 21:36 Models over harnesses: frontier vs. private models 32:33 Where AI wins in detection engineering 37:40 What Drew wants to build next 40:02 AI asymmetry: attackers with no guardrails 45:01 Ransomware chatbots and follow the incentives 47:57 Where to find Drew ABOUT DREW Drew Pilarski has spent 11 years in information security. He started as a self-described "tool monkey" doing generalist work, and for the last five or six years has focused on threat detection, including cloud threat detection. ABOUT PROMPT TO PROD Prompt to Prod is a podcast about building with AI, hosted by Aaron Mog. Each episode, practitioners from security and engineering share what they're actually building, what breaks on the way to production, and what they'd do differently. #PromptToProd #BuildingWithAI #AIAgents #ClaudeCode #Cybersecurity #DetectionEngineering #SecurityEngineering

  • 9 Episodes

About

Prompt to Prod is a podcast about building with AI. Hosted by Aaron Mog, each episode features practitioners from security and engineering who are shipping real things with AI: the tools they build, the workflows that stuck, and the lessons learned along the way. Expect honest conversations about build vs. buy, guardrails, data and model choices, and what breaks on the way from a prompt to production. No hype and no vendor pitches, just people who build talking about what actually works.

Information

  • Creator
    Prompt to Prod
  • Years Active
    2026
  • Episodes
    9
  • Rating
    Explicit
  • Copyright
    © 2026 Prompt to Prod
  • Show Website
    Prompt to Prod