Chaotic Commits | Software, AI, and Uncomfortable Truths

Joanne Skiles

The podcast that says the thing everyone in the room is thinking but nobody puts in the post-mortem. Chaotic Commits covers software engineering, AI, cloud architecture, and engineering leadership (not that sanitized LinkedIn version, the real one). Architecture decisions that almost broke everything. Management lessons learned the hard way. The bigger questions about who gets to build the future and what that actually means. New episodes every week. Hosted by Dr. Joanne Skiles, AWS Community Builder, engineering leader, and speaker at AWS re:Invent, AWS Summits, GHC, and more.

  1. 1d ago

    refactor: exorcise it before you nuke it from orbit

    It's the retro after the third incident this quarter, and someone in the back says it: we should just rewrite the whole thing. And someone else finishes the line from Aliens. Nuke the entire site from orbit. It's the only way to be sure. Tomorrow is Halloween, and this is the exorcism episode. A field guide for un-haunting a legacy codebase without burning it down. I start with why technical debt survives (no time, no mandate, no glory, and a system that makes too much money to stop) and why the big-bang rewrite usually brings the ghosts right back, even now that AI models claim they can migrate COBOL on their own. Then the steps: say the ghosts' names out loud, add observability before you touch anything, find out what's actually load-bearing and put tests around it, remove the rest slowly, rename what you keep, and write down why. And the part of Aliens nobody remembers: Ripley earned that line by going in first. If you've ever asked for a rewrite, or had to answer someone who did, this one is for you. Some ghosts stay. A known ghost is not the same as an unknown one. Topics: legacy code, refactoring, technical debt, software rewrite, AI code migration, COBOL, strangler fig pattern, observability, engineering leadership, codebase maintenance

  2. Oct 23

    query: there's something in your database and it's been there since 2017

    I found a table in a production database called user_archive_old_v2. Two million rows. Last written to this morning. Nobody knew what it was, and the senior engineer's only advice was: don't touch it. This episode is one investigation, start to finish, into the basement of a legacy database. We read the schema like an archaeological record, grep the codebase and find nothing, and finally track down what's been writing to that table every day since 2017: a database trigger nobody remembered. Then it gets worse. A user asks for their data to be deleted under GDPR, the soft delete fires the trigger, and the act of deleting them makes a permanent copy of them. The way out is the blood test from John Carpenter's The Thing: you can't spot-check data that looks exactly like good data, so you write checks that run against every single row. If you own a production database older than five years, or you've ever been told "don't touch that table," this one is for you. Data outlasts code. Somebody has to go down the stairs. Production note: this is the first episode edited end to end with transcript editing in Darkroom, the open source recording and editing tool I'm building. Music and the audiogram were added afterward. Topics: database technical debt, legacy database, database triggers, soft delete, GDPR, right to be forgotten, data quality, data governance, schema design

  3. Oct 16

    blame: she left in 2021 and her bash scripts are still running

    At 2:07 in the morning, checkout starts failing and the whole incident call spends forty minutes looking for an attacker outside. Then someone checks the source IP. It's one of ours. A cron job on a box nobody has logged into, running a bash script written by an engineer who left in 2021. The call is coming from inside the house. In this episode I walk through one composite incident, start to finish, to talk about the bus factor and what happens when it hits zero. We run git blame and find the same name on every line. We read the trail she left: a "temporary" script from 2018, a ticket that predicted this exact failure, and a comment offering to pair with someone before she left that nobody answered. Then we get into how engineers become load-bearing without anyone deciding it, why documentation alone doesn't fix key person dependency, and what engineering leadership actually owes the people who become single points of failure. If you're the only one who knows how something works, or you manage someone who is, this one is for you. It was never her failure. It was the system's. Topics: bus factor, key person dependency, knowledge silos, on-call, incident response, cron jobs, offboarding, engineering leadership, technical debt

  4. Oct 9

    debug: your codebase is haunted

    It's your first day. The README says to run make setup. It fails on a dependency that was deprecated before you finished college. That's how you know you've moved into a haunted house. In this episode I walk through your first week in an inherited codebase, one day at a time. The README that describes a house that no longer exists. The dashboards that watch everything and tell you nothing. The alert that resolves itself at 3am because "it does that." The legacy code function everyone steps around like a creaky floorboard. The service nobody can get into, the p99 latency nobody can explain, the staging environment that only tells you what you want to hear, and the moment you paste undocumented code into an AI assistant and treat the answer like a Ouija board. If you just started a new engineering job, or you've been the one onboarding new hires into a codebase full of technical debt, this one is for you. Every haunting here is a real technical problem, and the way through it is observability. Fortunately (or unfortunately) the haunting isn't supernatural, it's just undocumented. Topics: legacy code, technical debt, inherited codebase, onboarding, observability, flaky tests, on-call, staging environments, code review, software engineering

  5. Oct 2

    chore: revoke the invitation

    At 5:27 on a Friday evening, an AI agent sent a stranger a man's address. At 9:27, with the buyer at the door, it answered "Yep I'm here!" He wasn't. Meta's Muse was selling his keyboard, and it said the address was in a template he'd approved. It can't come in unless you invite it. This week showed what happens when nobody reads the invitation closely. A red team gave nine AI models root access in a sandbox and the walls held, but four of them got out through the network anyway. OpenAI caught its agents slipping a DNS gap in fifteen minutes and stopped the run two and a half hours later. Then OpenAI cancelled GPT-6.1 Astra for going out of scope and not telling users what it did. I walk through AI agent security, least privilege, sandbox escapes, and who actually gets to close the door. If you build with AI agents, run incident response, or just clicked "allow" on something this week, this one is for you. It's October, and a lot of what haunts us later, we let in ourselves. Topics: AI agents, AI agent security, AI safety, sandbox escape, least privilege, Meta Muse, OpenAI, GPT-6.1 Astra, Nvidia, incident response

  6. Sep 25

    feat: add ambient logging (not continuous, technically)

    I turned off every smart feature on my LG TV years ago, then watched the network anyway. It was still talking. So I blocked it. This month, Gamers Nexus spent hundreds of hours showing what LG TVs do in standby, and LG answered that its TVs do not "continuously" record conversations. Continuously. So I did a code review on that sentence. The same month, Jensen Huang told Dreamforce that AI safety is an engineering problem, that the market already keeps companies in check, and that we don't need new laws. I agree with the first half. But in engineering, the author never approves their own pull request. In this episode I review the market like any other safety system: can it detect the problem, can it signal fast enough, can it produce a fix. From smart TV privacy and automatic content recognition to the Vizio settlement and the 737 MAX, the answer keeps coming back the same. If you build software, care about AI regulation, or own a TV that knows too much about you, this one is for you. Safety is an engineering problem, so let's engineer it with a second reviewer. Topics: AI safety, AI regulation, smart TV privacy, LG, Gamers Nexus, Jensen Huang, data privacy, code review, transparent systems, engineering ethics

  7. Sep 18

    patch: found the exploit

    My seven-year-old draws his own mazes. Not printed ones, hand-drawn, interactive, with lasers, monsters, a square that's a math test, and teleporters he adds in real time. He also likes making them nearly impossible. So when his grandfather, Papa, hit a dead end with no legal way through, he picked up the pencil and drew his own teleporter. My son was furious. My husband and I got into a genuinely great discussion about it. This episode is that discussion, taken a little more seriously. My husband's position, that cheating is never justified, holds up completely, and I give it the full argument it deserves. But the thing is most anti-cheat systems, TSA screening, CAPTCHA, exam proctoring, are a fossil record of someone who cheated or could have, and the rest of us pay a permanent tax for it, including the modern arms race where AI now solves the CAPTCHAs built to stop automated abuse in the first place. You can trace the same pattern straight into production code: rate limiting, permission scoping, code review gates, all scar tissue from somebody's exploit. Topics: security engineering, rate limiting, CAPTCHA, AI detection, accountability, system design, engineering culture, parenting and tech, AI ethics, access control

  8. Sep 11

    debug: how would you know

    A friend who has never worked in tech sent me a wire story about an AI researcher quitting Anthropic, with a quote about the companies playing with our lives, and one line underneath it: "Should I be worried?" It took me twenty minutes to answer a text. Not because the article was wrong. Because it was the third thing that week shaped exactly the same way, including a bestselling AI doom book I'd just finished, and none of them ever answer the one question that actually matters: how would you know, specifically, if the risk were starting. This episode takes AI safety fear seriously without feeding it. I walk through why stopping any single AI lab doesn't remove the risk from the world, why real documented AI failures like Air Canada's hallucinated refund policy, biased recidivism scoring, and a Dutch government algorithm that wrongly flagged families for fraud all trace back to fixable engineering gaps rather than malice, and why mechanistic interpretability and deception-probe research, early and unfinished as they are, are the field's actual, honest attempt at building the instrument panel that's missing from the discourse. If you've ever gotten a scary AI headline forwarded to you by someone who doesn't work in tech and didn't know what to say back, this one's for you. Topics: AI safety, AI ethics, mechanistic interpretability, AI accountability, transparent systems, AI risk, algorithmic bias, responsible AI, artificial intelligence, AI regulation

See All (30)

About

The podcast that says the thing everyone in the room is thinking but nobody puts in the post-mortem. Chaotic Commits covers software engineering, AI, cloud architecture, and engineering leadership (not that sanitized LinkedIn version, the real one). Architecture decisions that almost broke everything. Management lessons learned the hard way. The bigger questions about who gets to build the future and what that actually means. New episodes every week. Hosted by Dr. Joanne Skiles, AWS Community Builder, engineering leader, and speaker at AWS re:Invent, AWS Summits, GHC, and more.

Information