Acima Development

Mike Challis

At Acima, we have a large software development team. We wanted to be able to share with the community things we have learned about the development process. We'll share some tech specifics (we do Ruby, Kotlin, Javascript, and Haskell), but also talk a lot about mentoring, communication, hiring, planning, and the other things that make up a lot of the software development process but don't always get talked about enough.

  1. Episode 101: Critical Thinking

    10h ago

    Episode 101: Critical Thinking

    This Acima Development Podcast episode connects a NASA document on critical failures to the modern temptation to over-rely on AI. The hosts open with examples of reward-function failures: a reinforcement learning agent in the game Coast Runners that racked up points by endlessly circling a lagoon instead of finishing the race, and an early GPT model that compulsively opened a calculator because its reward was dialed too high. These illustrate NASA's identified root cause of major incidents: lack of critical thinking and over-reliance on computers, processes, and procedures. The recurring metaphor is "playing with your head up," meaning you execute a strategy efficiently while continuously re-evaluating whether it's still the right one rather than blindly trusting the process. The middle of the conversation explores how much to trust AI, particularly for code review. The consensus rejects all-or-nothing thinking: AI is another tool, like a human reviewer, and catches different things, so the best approach is using both. Dave shares wins (AI flagging a subtle nil-returning bug he'd have missed) and failures (AI ripping out a carefully crafted Arel query to jam in raw SQL, or re-implementing a feature the team had deliberately killed). The takeaway is that AI can't run unattended because it lacks institutional lore, and an experienced human needs to stay in the loop, especially for high-stakes changes. Low-risk, templatized work like certain database migrations are good candidates to let AI handle, applying risk frameworks (mitigate, eliminate, transfer, accept) to decide where it can safely "cut its teeth." The discussion broadens into how skills evolve as technology shifts. Using analogies of welders replaced by robots, miners re-skilling into adjacent jobs, and the Pony Express being rapidly outpaced by the telegraph and railroads, the hosts argue that giving up certain "critical thinking taxes" (like manually smelting iron, or writing low-level compilers) frees people to build at higher levels, while foundational skills survive in niche "ranching" spaces like microcontroller programming. The unifying through line is proprioception, a visceral, grounded feel for what you're doing. Even as AI tools do more, you can't fully let go of the reins, because grounding in the underlying craft is what lets you sense when something is going wrong. Transcript: MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I am hosting again today. With me today, I have Eddy. I've got Dave.  DAVE: Howdy, howdy.  MIKE: Ramses and Kyle. It's possible we'll have a little bit of ebb and flow in who we've got here, as we sometimes do, which is great. We have a group here that can go into a good topic. Just a heads-up: we're going to talk a little bit more about the NASA document we've been talking about on the last couple of episodes. So, as usual, I'm going to start with some background story. This one, I'm going to go more on the technical side.  There's a great example, and you can look it up. It refers to a game called Coast Runners. So, there's a video game called Coast Runners. And some years ago, they built a number of reinforcement learning agents to play it. This is some of the early days of, "Hey, we can actually build AI that can play games, and it's working." Because there were a lot of years where they couldn't make that work very well, and it's kind of some of the first steps, "Hey, this is actually working." It led to some of the breakthroughs where we saw, you know, amazing things happen a few years back in video game playing, you know, including beating human players in a variety of games.  Well, this game, you know, it's not chess; it's not Go. It's a video game where you drive a boat, and you try to win the race. And they built the environment, and they built these reinforcement learning agents to play it. And not surprisingly, the reward function was to maximize your score. And here's where things get interesting. The score in Coast Runners is partially determined by winning the game, but it's also determined by how many of these little boxes that pop up that you can hit along the way. And what the agents learned to do is there's a lagoon that you can go into on the side. And...well, I'm going to read this. This is taken directly from a write-up that OpenAI did on it. It said: "The reinforcement learning agent finds an isolated lagoon where it can turn in a large circle and repeatedly knock over three targets, timing its movements so as to always knock over the targets just as they repopulate. Despite repeatedly catching on fire, crashing into other boats, and going the wrong way on the track, our agent manages to achieve a higher score using this strategy than is possible by completing the course in the normal way. Our agent achieves a score on average 20% higher than that achieved by human players, [laughs]" which is to say, computers will do exactly what you tell them to do.  DAVE: A follow-on that. In one of the...I think it was ChatGPT, in their training —the early ChatGPTs —if you'd ask a math problem, it would get it wrong because it was just doing words and tokens, right? It wasn't actually conceptualizing the numbers. And one of the things that they started reinforcing was open the calculator. If you get a math problem, open the calculator, type in some numbers, and add it, and when you hit equals, you get a cookie, right? We're going to increase your reward satisfaction score.  And they shipped this thing, and I want to say it was GPT-3, 20 to 40% of queries that you sent, no matter what they were about, it would silently open up the calculator, add one plus one, and hit equals, and give itself a cookie, and then go back and resume [chuckles] and answer your question. They had prioritized use the calculator instead of confidently guessing off of the words, right? Because AI's confidently [inaudible 03:42] the wrong way. They had dialed the reward up so high that it liked playing with the calculator. And it just reminded me of being a junior in, like, geography class playing [inaudible 03:51], you know, waiting for math class. Bless his little heart.  MIKE: Exactly, well, yes. And the reward will be maximized. So, NASA looked at some of the failures that they had had, and we've been talking about this document they published about a decade ago. And they said that one of the root causes of the key failures that they've had in some of the major incidents was lack of critical thinking. They refer to it as over-reliance on procedures and computer codes.  And there's a longer paragraph; I'll quote from this as well. Sorry for reading the quotes, but they're well-written [chuckles]. It's worth giving some context. "The third root cause is closely related to second. It is the lack of critical thinking with an over-reliance on computers, processes, and procedures. Computers, processes, and procedures are necessary but can never take the place of the human mind. In the end, all of our resources, tools, et cetera, are there as an aid to the human mind and the human creativity and decision-making." Now, this is fascinating a decade later, where we have AI becoming so much more successful, and we're outsourcing a lot more of our thinking. And if we rely on that without supervision, we get what we ask for. You know, like King Midas [chuckles], we may get exactly what we ask for and regret it for the rest of our lives. DAVE: There's a fun takeaway to this as well. I'm going to take this immediately and go meta, which is that, like, when I was in college, somebody barked at me and said, "Dude, you need to learn to play with your head up. Did you not ever play basketball?" And I'm like...I was not athletic at all. I don't even know...That's the round one, right?  MIKE: [laughs] DAVE: And, yeah, right? So, he said, "When you play basketball, what they teach you is don't look at the ball. You have to play with your head up because you have to be watching. You have to be looking at the net. You have to be looking at the other players. You've got to be able to dribble the ball without watching the ball." That stuck with me because he wasn't talking about dribbling the ball. He was talking about I was writing a piece of software that nobody in the room wanted. And one of the executives who had direct control over my paycheck directly did not want, and it was costing me dramatically politically. And so, basically, it was like, read the room, right? But the phrase "play with your head up" has stuck with me as, once you've decided on a strategy, you should execute the strategy efficiently. But you constantly have to be evaluating: Is this still the right strategy? When you find a good idea, it's very valuable; but also valuable is when you get rid of an idea that's no longer valuable —that's no longer earning its keep. And if we say, "Don't use AI because you won't learn nothing," we are at risk of falling into the very same trap that we just outlined because, for instance...I told this story on a previous podcast. I needed an auto clicker in Linux, and I know how to write one, kinda. I mean, I know the principles. I know how to write a timer. I know how to send input to the mouse. I know how to capture keystrokes. I know conceptually that I'm going to need to intercept a keystroke in another application, and that's going to require elevated privileges because that's technically a keylogger, right? And I don't know how to do any of that in Linux. 15 years ago, I could tell you from memory how to do it in Windows because I was a Win32 programmer for years and years. But I never learned how to do it under Windows, under the X11 system. I went to an AI, and I said, "I'm just playing Minecraft, dude. I want to AFK and grind some resources. I just want an auto-clicker, and apt search on open-source repo isn't turning up an auto

    43 min
  2. Episode 100: Normalization of Deviance

    Jun 10

    Episode 100: Normalization of Deviance

    This episode of the Acima Development Podcast centers on "normalization of deviance" — the pattern where small anomalies get repeatedly ignored until they cause catastrophic failures. Mike opens with the Space Shuttle Challenger disaster as the anchoring example: engineers warned that cold O-rings could fail, but their concerns were drowned out by schedule pressure and accumulated tolerance for small deviations. The crew connects this to the Columbia disaster years later, where the same organizational lesson went unlearned, and to NASA's own "Elements of Engineering Excellence" report, which lists not questioning anomalies as a major root cause behind their biggest failures. The conversation then wrestles with the tension between safety culture and velocity. Will pushes back on pure risk-aversion, arguing that heavy regulation has real costs and that tech's "move fast and break things" ethos has produced enormous value. Dave introduces the META framework (Mitigate, Eliminate, Transfer, or Accept) and contrasts NASA's culture with SpaceX, which celebrates blowing up unmanned rockets because the risk was already accepted and the explosion yields data. Mike reinforces this with an analogy from his kid's rocket-themed birthday party, where different risk levels (model rockets, sugar rockets, thermite) warranted very different safety boundaries — treating everything as maximum-risk would have obscured where the real dangers actually lived. The group lands on a key reframe: rather than trying to control everything, build a monitoring culture that instruments heavily, tests to failure, and pays attention to the signal inside the noise. The final stretch applies these ideas to current software practice, including AI-assisted development. Matt and Dave debate whether vibe coding will dominate production code soon, with everyone agreeing humans must remain accountable for what ships. Will gives concrete examples of normalized deviance developers live with daily: thousands of ignored compiler warnings (some of which are genuinely dangerous), bloated mobile web performance, and test suites nobody expects to run clean. He notes AI could finally make the ditch-digging cleanup work economically viable. Mike closes by tying it back to the opening theme: entropy is the default, letting things slide is easy, but flipping the culture toward actively watching the data is what prevents small deviations from becoming the next tragedy. Transcript: MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today. With me, we've got Will Archer, Dave Brady, and Kyle Archer. DAVE: Howdy, howdy. MIKE: I'm going to start with a story, as I typically do, actually two stories, but one funny and one not at all funny. I'll start with the funny one. My wife, when she was in her late teens, decided to drive with her sister to college. She wasn't going to college yet, but she decided to road trip with her sister to college. And they made sure the car was good the day before, had been doing some maintenance, and they cracked the case of the cooling fan for the engine. WILL: Oh. MIKE: So, when the fan was running, it was bumping against this cracked part of the case, so you can imagine the sound of that, not good, right? They actually took it to a mechanic and got kind of a loose sign-off that, "Yeah, well, this isn't going to make the car break, but it's going to sound terrible, and you should get it fixed soon," like, "Okay." And they drove cross-country [chuckles] with that thing rubbing the whole way. And what they did is they just turned up the radio, so full volume, full road trip. They drove for, like, two days [laughs] with the volume cranked up, just ignoring it. And she's told the story for years. It's funny, you know, everybody in the family laughs. You can just imagine, just turn up the volume, and the problem goes away. That is one way to make a problem go away. The other story is related to what we talked about in our last episode. And we're going to continue with the topic we talked about in the last episode, which is the Space Shuttle Challenger disaster, which happened in, let me check my dates here, '86. I believe this happened in... DAVE: '86? MIKE: '86. That's the date that I was remembering, so 1986. There it is: 1986. So, I actually looked this up. I read about it on Wikipedia. As a kid, I remember watching this [chuckles] in school, and it was, you know, horrifying. So, they had O-rings around the booster engines that, you know, like rubber or rubber-like material. And they had had record cold, I guess, at the launch pad the night before. And that cold caused the O-rings to, you know, shrink and stiffen. And so, in the launch, they lost integrity, so air started getting into the fuel. Eventually, that caused a catastrophic explosion, and the entire spacecraft disintegrated. I remember the horror of seeing those booster engines just randomly wandering, and there was not anything left of the main craft. It was a tragedy, you know, a horrible tragedy. Anybody who was around that time remembers. I was talking to somebody else like, "Oh, that was our JFK moment," you know? Everybody remembers that. Where were you when that happened? And it turns out the engineers had warned this might happen, and they were ignored, because there was enough noise in the data. They're like, "Oh yeah, well, there are so many things that can go wrong [inaudible 03:18] DAVE: Not just ignored, though, right? They were told to stay in their lane. MIKE: I think that's right. DAVE: If I recall. Yeah. They were told to be quiet, yeah. MIKE: The interesting thing about that is there was another space shuttle disaster some 20 years later or so, where Columbia broke up in re-entry. And the diagnosis afterward essentially said, "We didn't learn from the last time." There were likely problems that were pointed out by engineers, and there was just so much pressure to make this thing work that the concerns were ignored, and people died as a result. And in the document that we started talking about in our last episode, which is...certainly you can look it up yourself. It's titled...this was published by NASA titled Elements of Engineering Excellence. It was published in 2012. They made a list of root causes behind the major problems that NASA had had over the decades previous. Last time, we talked in depth about the importance of hands-on experience, that unless you have people who really have, you know, kind of gotten their hands in the work and understand it deeply, then you're going to miss stuff. The second point is what they call normalization of deviances. They also refer to it as not questioning anomalies. I'll quote from the report, "As was evidenced in the Challenger failure, we see deviations, and they're not quite normal, but seem to have no major consequence. After seeing these deviations a few times, we accept them as normal and ignore them. The result is a major failure where the deviation becomes catastrophic." So, that's our main topic for today, is the importance of questioning those anomalies and being able to see that signal inside, you know, a bunch of noise, because there's always noise [crosstalk 05:18] DAVE: There are a couple of interesting extrapolations on that as well. WILL: So, I have some thoughts about, like, sort of, like, these sort of, like, normalization of deviances and ways that it can go wrong. But, like, I suppose, like, and maybe this is just my priors, but I'm very much a believer in, like, a move fast and break things sort of ethos. Like, I'm familiar with heavily regulated, heavily controlled industries where, rightly or wrongly, there are high stakes, people die, right? And let me tell you right now that there is a cost to that. There's a substantial cost to that. And I do think that technology, in general, is pretty out of control in terms of, like, accountability, right? I mean, if you look at, like, you need a license to braid hair [laughter]. But, like, I didn't even need to graduate high school to do the job I'm doing. I just needed to convince somebody to give me a shot and then not get fired for long enough, and then you're in. DAVE: And we're writing software that handles people's money for them. WILL: Yeah, to the tune of billions of dollars, you know? And it's just like, "Yeah, you know, he sounded like he knew what he was doing. Let's roll," you know, which is fun. I think that's wrong. But, I mean, you can't argue with results of the industry that we've been in, right? And I think there's benefits there, and there's a lot of stuff where, yeah, you can let it slide until it blows up. You could do that. That's a strategy. It's a valid [crosstalk 06:56] MIKE: Well, and not only is it a strategy. It's a critical one. WILL: Initially, right? MIKE: Yeah. Well, absolutely. And even in regular life, you can't pay attention to everything. Attempting to do so would not end well, right? Our brain is very good at removing extraneous information. You can't pay attention to everything. So, you have to prioritize what you actually give attention to, and that better be the important stuff. DAVE: There's a rule in insurance, which is if you can afford to replace it, don't buy the insurance. But if you can't afford to replace it, don't even ask how likely it is that you're going to lose it. You have to get the insurance. The entire science of risk assessment is getting people to stop thinking about reducing the likelihood of a catastrophic fault and dealing with the case of when it is catastrophic, right? It's like, if you're going to take, "Oh, this hash collision can happen one in 10,000 times, and it will bankrupt the company," and I turn the PR back to you, and you say, "Okay, well, I've reduced the likelihood to one in a million times, and it still bankrupts the company." No, I'm not going to app

    43 min
  3. Episode 99: Hands-On Expertise

    May 27

    Episode 99: Hands-On Expertise

    This Acima Development Podcast episode centers on a NASA root cause analysis document from 2012 that concluded the agency needed to "reestablish the culture of technical excellence based on hands-on work." Mike opens with stories of Dutch Renaissance painters Rachel Ruysch and Maria Merian, both of whom spent decades honing their craft and improved continuously through sustained hands-on practice. This sets up the core question that drives the episode: does technical leadership require having done the technical work yourself? Dave kicks off the debate by asking whether the CEO should write code, prompting Will to share the "toxic and unpopular opinion" that technical executives should have built software at some point in their careers. The group largely agrees that hands-on experience matters, but the conversation gets more nuanced as it goes. Justin highlights how technically credible leaders create better engineering cultures because people can't pull BS on them, while Matt pushes back as the lone dissenter, arguing that communication, problem-solving, and trust in capable people matter more than personal technical skill, especially for CEOs versus CTOs. Will draws a distinction between line-level engineering managers (who he thinks should spend half their time writing code, ideally pairing) and higher-level managers who can step back. Kyle adds that genuine desire to understand the work can substitute for direct expertise, and Mike Porras connects it to decentralized authority, citing Atul Gawande's The Checklist Manifesto and the Katrina response failures as examples of why subordinate leaders need autonomy to act within their domains. The final third pivots to AI as a new threat to that hard-won technical grounding. Justin raises the concern that engineers are increasingly surrendering judgment to LLMs, which could produce the same erosion of expertise that NASA documented. Will frames LLMs as power tools and force multipliers, glorious in skilled hands but capable of taking your arm off, comparing them to rocket jetpacks. Mike Porras shares a more optimistic view, describing how AI lets him reach the PR stage faster on unfamiliar work and creates more space for higher-level architectural conversations with reviewers like Kyle. Mike closes by tying it back to the central thesis: the tool keeps changing, whether it's outsourcing, AI, or whatever comes next, but the need for grounded expertise, humility, and continuous learning never goes away. Transcript MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I am hosting again today. And I've got a big crew here today. I think we've got a topic we all care about [chuckles]. Here with me, we've got Eddy Lopez, Mike Porras, Dave Brady, Thomas Wilcox, Ramses Bateman, Matt Hardy, Justin Ellis, Kyle Archer, Will Archer, no relation [chuckles], Tim Chaffin, and Jordan Fong. Big crew of us here to talk about our topic today, and our topic today is triggered by a blog post by...her name is Vicky Boykis... Man: Vicky. MIKE: Yeah, Vicky Boykis. I ran into her from a newsletter, but, so I don't know her personally. But she wrote a fantastic blog about engineering, software engineering in the larger sense, machine learning, AI engineering specifically. Well, and interestingly enough, twice this week, I've randomly run into people writing about ancient Dutch painters. I say ancient, like, Renaissance Dutch painters, that era, and [chuckles], you know, women who were exceptionally skilled at their craft of painting during that period of time. And so, I couldn't just leave that alone. We're going to talk about that in our podcast [chuckles]. There's two women I read about this week. One of whose name...and these are Dutch names, so I'm probably going to butcher them. Give [chuckles] me some grace and patience on this one: Rachel Ruysch and Maria Merian. Both of them were Dutch painters. Rachel Ruysch was a painter of flowers who had painted, I guess, for decades, over a period when flower paintings were extremely popular, extremely sought after as a way in the long Dutch winter to bring a little bit of nature into your home. Maria Merian was not just a painter but a naturalist. She, as a child, was fascinated by nature and loved watching insects. And she carefully documented the process of metamorphosis, insect metamorphosis, which had not previously been documented well, meaning that many people in that era in Europe really didn't get it. You know, a lot of us as kids now, we think, "Oh yeah, caterpillar, butterfly. Got it." That was not the case back then [chuckles]. They did not connect the two. And we can thank Maria Merian for a lot of that understanding. She put things together that were not well understood for years. She actually, later in her life, traveled to South America and documented some of the amazing insects they have there, and people back in Europe didn't believe her. They just thought she was making it up [chuckles] because they just couldn't...you know, they were in Europe where it's cold, and the bugs are small. They just couldn't believe that there were these amazing tropical insects. And years later, her work has come to greater appreciation, that she beautifully captured and accurately captured things that the broader science community didn't catch up to for quite some time. One thing that both of these women had in common is they worked for decades, both of them starting, you know, like, in childhood. They were surrounded by mentors, and they worked, you know, they kept on working. You can find pictures of both women, because they were painters, right, paintings of them later in life, because they were experts. And here's the thing: they got better. One thing you do as a painter is you get better, right? So, their work improved over decades. It's part of why both of them are acclaimed today, is they just got really, really good from all this practice. Well, we're talking about Dutch painters, right? Let me connect this to one other thing before we jump in. About...well, I say about this many years ago. It doesn't matter how many years ago because you might be listening to this five years from now. In 2012, NASA published a document doing root cause analysis on problems that had led to some of the issues they had within NASA that led to deterioration of quality, accidents, that sort of thing. And they wrote a list of five items, and we may get to those as we talk, but then they summarized it. And here's the one-sentence summary that I've got. To prevent problems, NASA needs to reestablish the culture of technical excellence based on hands-on work. That's what I'd like to talk about today. We talked about the painters [crosstalk 04:49] DAVE: Like the CEO who writes code? MIKE: Possibly [laughs]. DAVE: Okay. MIKE: And that's an opportunity to dig in. Like, should the CEO be writing code? DAVE: [inaudible 04:57] I'm not challenging. Yeah. MIKE: Yeah, no. And I open it up. You know, I like to open things up and then be quiet for a while. So, you led out with that, Dave. Should the CEO be writing code? DAVE: I mean, not in prod [laughter]. When I worked at CoverMyMeds, that was the policy. If you work here, you write code. And so, there was a data migrator that was written by the CEO. And yeah, like, five years on, it read like code that had aged, had been written by an executive, and, you know, hadn't been kept up to date, and it was fine. The important thing was he had his fingers in the guts of the system, and that kept him a lot more present to how the system worked underneath him. WILL: I have a toxic and relatively unpopular opinion that, like, technical executive leadership should have, at some point in their careers, written software. I tell you, man, in terms of, like, CTOs that had the experience of working under possibly a third qualify, I don't know, I mean, that's, you know, I'm coming in with the hot takes. MIKE: I've had some conversations with our recently hired CEO. He is very much an advocate of engineering excellence and really wants to lean into engineering. He wouldn't have given it as a hot take, so...[laughs]. But he would have shared a similar sentiment because, in his feeling, if you don't have the background, then it's really hard to make the right decisions. WILL: So, yeah, 5 years ago, anytime [inaudible 06:37] 20 years ago. [laughs]. JUSTIN: I've been in a couple of places I've worked where the CTO has had technical excellence. The vast majority of the time, that has resulted in, like, a better engineering culture than, you know, those without. And by that, I mean, like, engineers wanted to stay because this was...yeah, not like now, but this was during the time when it was hard to hire engineers. And you spent a lot of thought about how to make a good engineering culture. If you did not have a good engineering culture, people just left, and they weren't happy there. And having a technical leader in technical leadership was key to that because they just...they got it, and you weren't able to pull BS on them, or they had less BS pulled off on them because they've been in the trenches. And those who haven't been in the trenches, they're up in their...I wouldn't even call it an ivory tower. They're off in their palace somewhere doing politicking, and politicking always sucks [laughs] so... MIKE: You know, Dave, I think you asked at the beginning: should the CEO have written code? Well, maybe, maybe...So, here's a take on that. Our CEO currently was the CFO before, so he's deeply experienced with finance. He comes from a background where he knows that. And I probably trust him more as CEO because of that, because I know that he knows where to, you know, watch the money, and he's not going to let things get out of hand. WILL: It's a financial company, right [laughs]? You know. MIKE: Yeah, exactly. It's [laughs] a fin

    43 min
  4. Episode 98: Standups

    May 13

    Episode 98: Standups

    This episode of the Acima Development Podcast starts with a discussion about the frustration of U.S. tax filing and uses it as a metaphor for poorly run standup meetings in software development. The hosts argue that many teams repeat painful, unnecessary processes simply because “that’s how it’s always been done.” From there, they unpack the most common standup failures: meetings turning into status reports, running too long, involving too many people, or becoming impromptu debugging sessions where only a few participants are engaged while everyone else checks out mentally. The panel emphasizes that these problems are usually symptoms of poor communication and coordination happening outside the standup itself. A major theme throughout the conversation is that standups should focus on coordination rather than status reporting. Dave Brady argues that if teams properly maintain tools like Jira or Kanban boards, everyone should already know the project status before the meeting begins. The standup’s real purpose is identifying blockers, avoiding collisions between teammates’ work, and quickly coordinating handoffs. The hosts debate alternatives like “Slack-ups” and asynchronous updates, with some arguing they fail to replace the human interaction and spontaneous coordination that happens in live meetings. They also discuss ideal team size, meeting frequency, time zones, and how distributed teams create additional coordination challenges, especially when work is handed off between regions. As the conversation evolves, the podcast becomes less about standup mechanics and more about human connection in remote work. Will strongly advocates for cameras and microphones being on during meetings, arguing that face-to-face interaction helps managers recognize burnout, disengagement, or personal struggles that text updates can easily hide. The hosts criticize workplace cultures that dehumanize remote or offshore workers by treating them as interchangeable resources rather than teammates. By the end, the group concludes that the biggest failure in bad standups is not inefficiency alone, but the loss of genuine human connection. Good standups, they argue, are ultimately about building trust, communication, and healthy relationships within a team, not simply exchanging status updates. Transcript: MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I am hosting again today. With me, I've got a great panel. I've got Thomas Wilcox, Dave Brady, Justin Ellis, Eddy Lopez, and Kyle Archer─I think we're all returning crew here ─[chuckles] to talk about our topic today. So, you're probably not listening to this, like, exactly when we're recording it. You're probably not even listening to it right when it comes out. There's always a recording period and then a publishing, you know, a week later, or a few weeks later, after it's gone through editing. And we have a bit of a queue in case we miss some time. It all works out. But we are recording this in tax week in the U.S. This was the week that taxes were due, and everybody has hopefully completed their annual suffering and has submitted those numbers to the IRS. I read about this before, and I read about it again this week. Articles are often published this time saying, "Why do we do this?" Well, it's a good question, because United States is actually fairly unique in the world in that we have to submit all these taxes every year. In many countries, most people don't have to do anything at all because if you're working for an employer, they've been submitting tax information to the government all year, right? They've been paying your taxes, and as long as you don't have anything funny going on, that's enough. The government knows about you, you know, they probably know how many dependents you have, you know, you've reported that. I mean, you reported it with your business. The information's there. And in much of the world, people just receive a letter saying, like, "Yeah, thank you. Everything's good." And they receive, you know, there's no refund or non-refund because it just works, right? They don't have to do anything. The cycle that we go through of pain every year doesn't need to happen. Now, the reasons for that have to do with...Well, I want to be careful here. Our purpose here is not to criticize large corporations who lobby heavily [chuckles] to keep the tax code as it is, well, to keep the tax submission process as it is. But such is our life, right? But where I was going with this is that we go through all the suffering because it seems normal, and everybody we know around us do it because it seems normal to go through all of this process of reporting something we've already been reporting with every paycheck for the entire year. It's just a rehash that we have to do in excruciating detail because that's how it's always been. But there are examples of people who do it differently, and they don't go through the same pain that we do. And I imagine that in their blissful lives, they have extra time around this season to do things other than pay their taxes or, you know [inaudible 03:14] DAVE: Must be nice. MIKE: It must be. Why do I talk about this? Most of you, if you're an engineer, have probably been in a lot of standups, which is a sometimes daily, sometimes weekly, some regular interval typically meeting where you have a chance to touch base and connect with other people on your team. And they can range from actually pretty good to something far from that [chuckles] to something that makes you want to quit your job right [chuckles]? Like, well, not another standup. The idea, you know, comes from this agile process where...and I think it's not even just engineering. You get together in a room. You want the meeting to be so short that nobody sits down, right? You go through the key things to make sure that everybody can touch base. Now, we have all kinds of communication channels, right? We've got, you know, our messaging platforms that we use. We've got the ability to go and walk over to people. There's lots of ways to communicate, but we decided that we're going to pay this cost of bringing a whole team. And this can happen at lots of levels. You can have executives getting together for a standup. You can have the team that reports to the executives getting together for a standup. So, you have a bunch of people, and that's an expensive meeting, right? Imagine the executives getting together for a standup. I don't know how many dollars that costs, right? I'd have to do the math, but it's not few. It's an expensive meeting where people have chosen to do that because they think the coordination is so important. But it can be done right, and it can be done wrong. It can be a yearly suffering, a period of suffering, like the taxes, that reports stuff that's already been known. Or maybe it's a meeting that ends quickly and touches on key information that not everybody knew because it was late-breaking, and it was a good opportunity to share. We're just going to talk about standups. It's something that we all live with, so it's worth talking about. So, I'm going to ask─I've given the intro─what have you seen? Well, actually, let me start. Let's start with the bad side. What is it that makes a bad standup? DAVE: Turning into a status meeting, for me. The thing that makes a standup go bad...and I will reveal the point that I wanted to make in today's podcast right out of the gate. The thing that makes a standup go bad is when you are not taking care of the things that you need to take care of outside of standup, and so they have to get taken care of in standup. When your standup runs really, really long and turns into a gigantic status meeting, it's because you're not communicating status outside of the meeting. And, actually, I don't need to put any more on that point. That's just like, if you don't take care of it elsewhere, it's going to hit here. When I look at a standup that's running long, I don't look at it as, like, this meeting is bad, I mean, it kind of is. But I look at it as, okay, what is the unmet need that is screaming at us so loudly that it's cratering our standup meetings? That is frequently a very helpful thing. If it's a status meeting, you maybe need to, you know, do better in, you know, one of your other practices. If you're arguing about cleaning up code, maybe your retro needs to be better. Yeah, that kind of stuff. MIKE: Okay. So, you said there are some other venues where this should be happening, that the status reporting should not happen in standup. I've been in a lot of standups that were about status reporting, so, you know, you're bringing up a common failure case. If that's the bad case...well, I want to come back to [inaudible 06:46] JUSTIN: There's more bad cases. We got more [crosstalk 06:50] DAVE: We should go through the counter good case to the status MIKE: Yeah. So, let's go to the other bad cases, but let's put a pin in that one because you said, status report: bad [inaudible 06:59]. So, what are the other bad cases? What are other bad cases around standup? JUSTIN: When they run long, and there's not a good reason for it. I mean, basically, when you go back to your summary, you talked about how everybody is standing up, and they don't want to, like, sit down, and you just want to quickly go through things and be done. If it's going more than 15 minutes, or it's going more than 20 minutes, whatever you have allocated, and it shouldn't be more than 20 minutes probably, that means everybody's looking at their watch. They're wondering about what other meetings they have to go to. They aren't focused. And you, all of a sudden, the only person who is paying attention is the person you're talking to directly, and everybody else's mind is just like, pshhh [laughter]. DAVE: And you're 100% guaranteed at that point...if your meetin

    56 min
  5. Episode 97: Database Indexes

    Apr 29

    Episode 97: Database Indexes

    The episode of the Acima Development Podcast centers on database performance, using the concept of indexing as its foundation. Mike opens with a story about discovering Google in the early 2000s to illustrate how powerful indexing systems transformed access to information. That same principle applies to databases: indexes act as shortcuts that make retrieving data dramatically faster, especially in large datasets. The discussion emphasizes that while indexes can feel like a technical detail, they are fundamental to how modern systems function efficiently, much like search engines reshaped how people find information. Bill Coulam then dives into the technical side, explaining that indexes improve read performance but come with trade-offs, particularly slower writes because both the table and index must be updated. A key rule of thumb is that indexes are most beneficial when queries return a small subset of data, typically under about 25% of rows. The group explores how poor indexing strategies, like over-indexing or missing indexes on key relationships, can quietly degrade performance over time. Bill shares a striking real-world example where adding missing indexes reduced a process from taking 24 hours per record to processing millions in just a couple of hours, highlighting how impactful proper indexing can be. The conversation broadens into database design philosophy and performance tuning. The team discusses different index types in PostgreSQL, when to use them, and how to balance read vs. write performance depending on use cases like bulk inserts or high-frequency queries. They also touch on when relational databases fall short, such as full-text search or massive write-heavy workloads, where NoSQL or specialized systems may be better suited. Ultimately, the takeaway is that effective database performance comes from understanding your data, access patterns, and trade-offs, combined with ongoing maintenance and thoughtful design rather than relying on defaults or assumptions. Transcript: MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today. I'm going to start by introducing Bill Coulam, who's with us today. He comes to us from the data team. And he's been here before, but we're going to focus on some information that he has to share. So, he's kind of the star of the show today. Also with us, returning, we've got Eddy, Travis, Justin, Dave. Mr. Perez, great having you with us. We've got Mike Perez here with us, and Ramses. As usual, I'd like to start with something a little bit outside of our topic in order to bring it in and tie it into the outside world. And I was thinking about a story I think I've shared before. The importance of this moment early in my career keeps, like, growing as I look back to it, like, wow, that was a big deal, and I didn't realize it at the time. So, in the early 2000s, somewhere in the early 2000s, early, early 2000s, I was working for a guy [chuckles]; I'm going to say that. He had some projects, and he didn't have enough resources to do some freelance projects, and so I was doing some of his stuff. He was outsourcing his freelance work to me [laughs]. And he had a project that was in Windows, and there was something they wanted to accomplish through the API. And I started looking through the documentation, trying to use Microsoft's tools to search the documentation, and I spent hours. I looked everywhere I could [chuckles], and I couldn't find it. I came to the conclusion, maybe this doesn't even exist. And I came back to him, and he got back to me, like, 30 minutes later. He said, "You know, there's this new tool called Google, and I use that, and it's amazing. You should start using it because it works really well, and it led me to this documentation." Like, wow, well, I know what I'm going to use now. I'm going to use this Google thing [laughs] because that works way better than actually going through the table of contents, and the index, and the documentation, because that's really hard to search through. Those older forms of indexing were insufficient. Now, Google had this brilliant idea, you know, the founders of Google, that, okay, we'll index the internet. And even back then, that was, like, an impossible goal [chuckles]. And there were other sites that were doing it. There were indexes out there. What they would do is they'd look at the words on a website, and they would create an index based on those. And so, if you look for a word, they'd look for a website that had a lot of those words. Well, people really quickly figured out how to game that [chuckles], and, of course, they did. So, they were useless almost immediately because people would go into their meta tags, and they'd just write the same word a hundred times for something that the site was really not very applicable for. What Google did is they came up with a different sort of index, where they would index words in the links that linked back to a site, and also give extra weight if there were a lot of them, right? And so, by building a more appropriate index that suggested popularity, rather than self-determined, a self-stated importance of the page for a specific topic, they were able to come up with something way more effective. And you don't always think about indexes, you think, index? Like, I remember going to the library. It had, like, the Dewey Decimal System, which is really kind of weird and awkward and hard to find things with, but it was way better than the alternative, which would have been nothing. You don't usually think about indexes changing the world, but that index, that PageRank index, you know, the PageRank algorithm that they use to just create an index, that's all it is, right? Link this word, map this word to a website, so that when you're searching this word or phrase, then we can find it. It literally, like, fundamentally changed culture. It's now a verb [laughs]. Like, you Google something, even if you're using Bing for those of you out there who use Bing [laughs]... DAVE: Use Bing to Google, yeah. MIKE: Exactly. You use Bing to Google, because information now is accessible, and that is something that didn't exist before that. For all the digital natives who've grown up in this world, like, how did you find things before? Well, you didn't [laughs]. You suffered. You wandered through libraries. DAVE: We just got used to not knowing things, yep. MIKE: Exactly. That's exactly what you did. You got used to not knowing things. It changes everything when you have an effective index. And I could talk about all the times in my career when something's missing from the database, and yeah, it was the index. It's always the index. There's always a missing index somewhere. It solves all of your performance problems. And there probably is an exception, but I can't think of it [laughs]. It's always the index. That's what we're going to talk about today. We're going to talk about database performance. And we've been wanting to, you know, Bill's been preparing this and thinking about this for a while. If we're talking about database performance, indexes are going to come up over and over again. And this could seem really dry, and this is going to be a technical deep dive, right, we're going to very much going to talk about indexes. We're probably going to be focusing on PostgreSQL. But this idea of indexes is not a trivial one. It's how we operate in the modern world. Our culture, our commerce has been fundamentally transformed. Our ability to know things and outsource, you know, to this Library of Alexandria that we've got in our pockets all depends on indexes, and it's amazing. There's my introduction, Bill. And I wanted to lead out with some weight behind what you're going to be talking about today. BILL: I love it. That was a fantastic segue. All right. Hi, everyone. I am Bill, Bill Coulam. I've been doing this work for about 30 years now. I started as a software engineer using COBOL and mainframes, but I don't put that on my resume because I don't want anyone to ever call me back to help with that. So, I tell people I started with C and C++. I was actually one of the first users of Java back in 1995. My company that I worked for at the time, Anderson Consulting, they wanted me to go around to their clients and tell them what I thought of Java. And, at the time, I felt like it really wasn't ready for primetime, and so I kind of voted myself out of working on that platform. But that's okay because I ended up, on every project that I worked on, working with Oracle, and, at the time, Oracle was the 800-pound gorilla. And I was in the telecom industry, where we had some of the largest volumes of data in the world, and so I learned a lot of great lessons working on those big systems. It's a whole other world jumping between databases that have 10,000 to a hundred thousand rows to databases that have 500 million, a billion. Performance tests in your copy of production can take three hours. It's a completely different world. Anyway, so you learn a lot of good lessons working on data that big. I ended up sticking with Oracle for a long time. It became my bread and butter. And went from San Francisco to Denver to Houston, and then back here to Utah where I grew up. I've been here longer than I spent time in my own hometown. So, I've been here in the northern central part of Utah since 2007. Anyway, let's go ahead and jump into it. We're going to be talking about four areas: the fundamentals of indexing, some guiding principles, the two shared tendrils, index types that are available to us using Postgres as our source database, and some indexing dos and don'ts. Firstly, some fundamentals. An index is a shortcut to get at the data. However, because an index is a separate structure from the actual table containing the data, it requires at least two I/Os to get at the data: on

    59 min
  6. Episode 96: AI & Code Reviews

    Apr 15

    Episode 96: AI & Code Reviews

    This episode explores how AI coding tools are changing the role of code review. The hosts point out that AI can generate large amounts of code quickly and even review it, which shifts the bottleneck from writing code to reviewing it. While AI can handle repetitive or low-risk tasks like documentation updates or simple refactors, it can also produce inconsistent feedback and get stuck in loops. Because of this, teams need clear rules and priorities, such as focusing first on whether code works, then on security and performance. AI is useful, but only when its boundaries are well defined. The group discusses different ways to structure AI-assisted reviews. Ideas include using multiple bots to score changes, setting strict allowlists for what AI can approve, and blocking sensitive areas like business logic or database changes. They compare AI to a junior developer who can help but should not be fully trusted without oversight. Risk becomes a key factor, similar to self-driving cars where automation works best under specific conditions. Some participants prefer AI as an assistant that gives suggestions rather than one that approves code, since human judgment is still needed for context and decision-making. The conversation also highlights what is lost when humans are removed from the review process. Code reviews have traditionally been collaborative and educational, helping developers learn and improve through discussion. AI removes much of that interaction and can even create false confidence by being overly agreeable or flattering. This can lead to mistakes making it into production. In the end, there is no clear solution. Teams need to balance speed with caution, use AI where it adds value, and keep humans involved to maintain both quality and the collaborative nature of building software. Transcript: MIKE: Hello and welcome to another episode of the Acima Development Podcast. I am Mike, and I am hosting again today. With me, I have, as usual, Will Archer. We've got Thomas Wilcox. We've got Eddy Lopez. Dave Brady. DAVE: Hello. MIKE: [inaudible 00:35] join. And we've got, after a long absence, Tad Thorley [laughs]. TAD: Yeah, thanks for inviting me. MIKE: We bumped into him this week, and he came and joined us, so it's great to have you, Tad. And Tad actually kind of seeded our topic for today that we'd like to go into. As usual, I'd like to, you know, connect this to real life. I went fishing for a compliment today [laughs]. I was talking to my daughter at lunch time, and she was saying something to my youngest. I didn't even hear what she said, but she said something like, "Oh, because you're strong and tough." And I didn't know who she was talking to. And I said, "What was that?" She said, "Oh, I was talking, you know, I was talking to him." I'm like, "Okay, because I know that I am, you know, weak and fragile." And she looks at me [laughs], and then she says, "You are not weak. You are strong," something [laughs] along those lines. I thought, ah, thank you [laughs]. Thank you. Say nice things to dad. And I totally dug for that. Totally not deserved in any way [laughs], but I took it anyway. As humans, we like somebody to say something nice to us. It's always a good thing. But we also are totally prone to flattery. And [laughs] if somebody says something nice to us, we will believe it, whether it's true or not. Actually, this morning, early, I read a crazy story. Crazy story. And I'm not going to go into it in depth, but it involved a scammer in Mexico convincing a variety of U.S. movie executives to make a movie out of his story of being imprisoned by the Mexican cartels to play flag football [laughs]. DAVE: Flag football. That's the interesting-- MIKE: To the death. To the death. DAVE: To the death. Oh yes. Yes. MIKE: But, you know, you can keep [inaudible 02:21] WILL: But no contact until you die. MIKE: Exactly [laughs]. WILL: You're only going to take one tackle, but it's going to be a doozy. MIKE: I think they weren't allowed to tackle, but they were, like, breaking each other's teeth. And then if you lost, they took you out back with weapons, yeah. DAVE: It's a high lie. It's traditional down there. MIKE: [chuckles] It was a crazy story. Well, no, it was a scam artist who was pulling all this off from the beginning. But, you know, you can pull off a lot by just being really convincing and saying nice things to people, telling them what they want to hear. We'd like to talk today about code reviews [chuckles] and doing evaluations of human output. And we're in an interesting time period. A couple of years ago, even a year ago, maybe even six months ago, we would not have had this conversation. But there are tools out there now that can read your code and actually give pretty good reviews most of the time. In fact, in some ways, they're going to be better, and that "in some ways" is doing some work here. So, let me be clear: in some ways, they're going to be better than human reviewers. That is not universally true, I don't think, at this point. In fact, I think it's far from universally true, which brings us to our topic today. What does it mean to do code review today? There are tools that can do code reviews. What do they do well? What do humans do well? What does it mean? And we've talked before about code reviews. I think it's been a while. I think it's been maybe a year or two since we've talked about code reviews, the value of code reviews. So, we'll maybe touch on them maybe a little less this time. DAVE: And it was entirely a soft skills discussion, right? MIKE: Yeah, I think it was. I think it was. DAVE: Humans talking to humans. MIKE: Humans talking to humans. And now we've got the machines talking to the humans, and the humans talking to the machines, and the humans talking to the humans about what the machines are saying. It's totally scrambled. So, revisiting this idea of reviews with AI in the mix, now, Tad, again, prompted this discussion because he's been playing around with this and has found some solutions to some of the cases that go wrong [laughs]. There are degenerate cases where the AI will recommend that you change something, and then when it sees your changes, it'll recommend you go back to the way you were before [laughs]. If you're anybody who's used a linter, you've probably seen the same thing. It tells you to fix it, and then you cause a new problem. So, which one do you choose? That's where we get into art. That's not an unsolvable problem, but there are some interesting solutions there. Nor is it nearly the sum of all of the problems here because there are all kinds of edge cases here with reviewing with AI. With that introduction, Tad, I'm really curious for you to give us a little talking to about what you've been working on and some of the solutions you've found. TAD: Okay. Yeah. I just was mentioning something to Dave because I think what's really hard is I find that, with AI, I do way more code reviews than I've ever done before. And I was giving Dave an example because I can, like, just with my Claude Code setup, I was able to integrate it with Sentry, which is error tracking, and Linear, which is our task management, and GitHub, right, has a command line. And so, I could literally, with a prompt, say, "Look at our past 20 or so Sentry errors. Create Linear tasks for each one. Create a local work tree for each of those Linear tasks. Fix them in parallel in those work trees. Create a PR for each one, and assign Chris for every PR. Do that in parallel with subagents." And, for me, typing that up takes, I don't know, a few minutes. And now I've just given Chris, like, two days' worth of reviews, possibly, or something like that, right? Like, so much code could be generated so quickly and so easily that I find that the code review step is the biggest bottleneck. It usually is the bottleneck, but now it's multiplied. Like, it is absolutely the biggest bottleneck in the whole process. And I don't honestly know, like, a complete solution to that. But something that we were doing at work was actually bot reviewers, where we would say, you know, like, if your review looks safe enough, the bot will just approve it. And that was kind of an interesting experiment that we were doing where you have to -- But, like you were saying, Mike, one of the first issues that I ran into when the CTO kind of implemented that was I pushed up a PR, and it said, "This code is inefficient." And I'm like, okay. And so, I just had my Claude just keep checking GitHub and say...I told it every time it says there's a problem, fix it, and push up the fixes, and just do that until everything is approved, right? And my Claude Code, for about 45 minutes, tried that. And it kept flipping back and forth between like, "Oh, you're not doing enough security checks.” Oh, "This code isn't performant enough.” Oh, "It's not doing the security checks," and just back and forth in a loop. And my Claude Code, I could almost feel its frustration in its final message to me. It essentially said, "I cannot get a review past the reviewers. I keep going in this cycle, and they are never going to review this," and it just gave up [laughs]. And I'm like, wow, I've never seen a bot just straight up give up before, but here we are. So, yeah, like, that was our first, like, test of that. Our setup was, we had what we called the bot committee, where we had a Codex, and we had, like, a Claude Opus that would both review independently then, like, an aggregate score would be kind of brought together. And if the score was over a certain threshold, then it's like, okay, yeah, you can auto-approve this. But what I did last week was, I found I had to go in and be very clear in what was okay to pass and what was not, right? Like, you're updating some documentation; that's great, you know. You shouldn't have to have a human, like, approve you

    43 min
  7. Episode 95: What Do Data Engineers Do?

    Apr 1

    Episode 95: What Do Data Engineers Do?

    This episode explores the role of a data engineering team within a company and how it differs from traditional application development. While app developers focus on performance and real-time systems, the data team is responsible for collecting, syncing, and organizing data from many sources into a central warehouse (like Snowflake). Using tools such as Fivetran, data is continuously pulled from dozens of systems and stitched together into a unified view that business users, analysts, and dashboards can actually use. A major challenge discussed is how microservices (great for engineering) create fragmented data that must be carefully reconstructed to tell a complete story, such as the lifecycle of a customer or lease. A large portion of the conversation focuses on “data transformation,” which is the process of turning raw, scattered data into meaningful insights. This involves complex pipelines of queries and scripts that combine, clean, and interpret data across systems. The speakers emphasize that this work is far from simple—it requires deep understanding of both the data and the business context. Done well, it enables decision-making (like tracking revenue trends or customer behavior), but done poorly, it can lead to incorrect conclusions that impact the entire company. They compare transformation to cooking or even building a rocket: the output is fundamentally different from the raw inputs, and small mistakes upstream can cascade into major issues downstream. The group also discusses practical challenges in data modeling, system design, and collaboration between teams. Topics include the tradeoffs of normalization, handling schemas across evolving systems, and frustrations like poorly defined enums or lack of communication when engineers change databases without notifying the data team. Security is another key theme, especially around controlling access to sensitive data (PII) and preventing misuse. Ultimately, the episode highlights that data work sits at the center of the organization: it depends on upstream engineering decisions and directly influences downstream business outcomes, making clear communication, documentation, and thoughtful design essential as systems scale. Transcript: DAVE: Hello and welcome to the Acima Developers Podcast. We've got a fun group today. I've got Eddy. We've got Kyle. We've got Thomas. We've got Mike and Justin. We've got Bill, and we've got Zach. Now, Bill and Zach are infrequent. Bill's our DBA, and Zach is the...what are you? The head of the data team? ZACH: Technically, my title is Senior Manager, Data Architecture and Governance. But that's a fancy way of saying that I am heading up a data engineering team. Yep. DAVE: They made you widen the column size to fit that job title in. ZACH: Yeah, pretty much. DAVE: Yeah. Yeah. So, for people that don't know, I've been at Acima for almost five years, six years. I don't keep track of numbers. I worked in engineering for a couple of years, then I went over to work with Zach on the data team for a year. And then he got rid of me and sent me back to engineering. And I've been back over here for, like, a year and a half now. And I think it's really, really fascinating the different ways the teams work. Like, app dev focuses on latency, and we love to do everything with compute, and we're very scarce with storage. And the data team is kind of the other way around. You've got the great big warehouse. Storage is free. Compute is crucially expensive. It's like, you've got a table that has all the integers in it, and you look them up by ID because you can't calculate anything. That's a joke. But people don't believe me when I tell them you have a day’s table that is literally every day from 1970 forward. We don't want you to calculate the name of the day of the week. Just look it up in the table. We don't want you calculating the first letter of the day of the week. That's a separate column in that table, right? ZACH: Yeah. I don't think that that table was originally built for that reason specifically. I think a lot of people used it for that reason. There's a lot of really good days logic built into, like, Snowflake, Redshift, and all of the warehouses. However, when Acima first started, warehousing was a little bit newer, and so maybe a lot of those functionalities didn't exist. Now it's more like, what's a holiday [laughs]? And that's the main reason we're using that table is, what is a holiday? And that table is not always the most accurate on what a holiday is, either. But it's way more accurate than if we didn't use it [laughs]. And it's a data source that my predecessor exported from somewhere a decade ago and runs all the way through, like, 2060. So, I'll probably never adjust it, you know. It’s just -- DAVE: That was going to be my question, so when do we even run out of days? ZACH: It doesn't matter to me. It'll be long after I've, you know -- EDDY: Is that only taking into account local holidays, or now that you're considering, like, international growth, like, does the table also consider international holidays, or is it only local? ZACH: It's not been updated to consider international holidays. We don't have to do a ton with holidays on the data team. Really, that's going to be on our production systems, right? Like, we are consumers of data. We are not...Well, I mean, we generate data, too, but we're mostly consumers of data. If you look at the flow in, it's mostly data coming in. So, it's really important for, like, LMS to understand what a holiday is in every single country that they're in. Not as important for the data team because the events that should not happen on holidays, there should be no data for because they didn't happen, right? But no, I've not expanded that table for, like, Mexico or Canada or any other country. It's just U.S. And even then, like I said, it's not fully accurate. DAVE: I remember when I started here, we had no plans to go outside. We were just U.S. company, and so don't worry about it. And businesses pivot and grow. Zach, I got a question for you. I jumped straight into some detail, but I don't think a lot of people know what a data team does. We were talking about this in the pre-call. Like, the DBA does the architecture, but you guys...you said CrossFit. I work on Merchant Portal. My job is to help keep the merchants happy so that they can give leases to customers and get the product out the door. That's an application database written in Postgres. Where does my data go after, you know, like, every night, what happens to my data? What do you do with it, and who do you give it to, and what do they do with it? ZACH: Yeah, so that's a loaded question. Every 15 minutes, it syncs to the warehouse. We use tooling for that. That tooling is Fivetran. They're a great company. They have a bunch of people like me and smarter than me focusing on just, how do we sync data from data source to Snowflake or Redshift or a data destination, basically? So, it's the best way, in my opinion, to sync it. We used to have an in-house solution. It would miss data. We didn’t focus on it a lot because we have a bunch of other stuff. So, now it syncs into the warehouse. And especially in a system of microservices, which I know are great for software engineers, they're terrible for data engineers because the next piece of the puzzle is I have to stitch all that data together. A lease record, for instance, or really any record, is not going to be wholly in one service. So, now I need to create transformation tables so that our business users, our end users, our BI analysts, and the people viewing their dashboards can see the holistic view of the lease. Because, as you know, there's a certain point where Merchant Portal just doesn't care about it anymore, and it moves on to LMS. And then LMS doesn't necessarily care about all the nitty-gritty of what's happening behind the scenes in all the other microservices for, like, payments or anything like that. So, we really become the place where we're stitching that together. In the last count I had, I think there's 68 Postgres databases syncing into the warehouse today. DAVE: Wow. ZACH: We do not care about all of them [chuckles], to be frank. We do care about around 30 of them, and we use them for transformations. And then there's a bunch of just, like, batching, right? Like, I don't want, and you guys don't want, nobody wants the production customer-facing services spinning up jobs in the middle of the night to grab thousands or hundreds of thousands of records to throw them in a CSV and shoot them off to, like, a company that needs that information, right? Like a third-party company, maybe that we integrate with. And so, the last time I recorded, there was something like 50 third-party integrations that we're also handling. That data will go into those companies; data's coming out of those companies. Maybe the data goes into those companies in real-time events through the production consumer-facing services, but I am siphoning them into the warehouse so we can start to see, like, is this third-party company worth using? What is the effects that we are having here? Or maybe those companies are enriching our data, and then we look at that on the back end, and we let that adjust business decisions. And so, all that's got to come together in a singular place. And it's a lot. Like, the last time I checked, it’s...I keep saying, “Last time I checked,” I don't watch this like a hawk. But we had, like, 13 and a half thousand tables in the warehouse. So... EDDY: So, Zach, you mentioned something interesting, and I kind of want to elaborate a little bit. So, you said you have about 60-plus tables that have data, but you only care about half of them. What's the point of us -- ZACH: 68 schemas. So, like, Merchant Portal is a schema. Merchant Portal has, like, 218 tables. I care about those

    1h 3m
  8. Episode 94: Staying Cool During Production Issues

    Mar 18

    Episode 94: Staying Cool During Production Issues

    Mike opens by framing “production incidents” with a vivid non-software story. As a teenager he smashed bathroom tile with a dead-blow hammer, drove his pinky knuckle into a jagged shard, and had to manage both the injury and the panic of his little brother who got sick from seeing it. He uses that as the metaphor for on-call life. Bad things happen, reactions vary, and what you do in the first moments matters, especially staying calm, reassuring others, and focusing on the most urgent next step. The group riffs on modern incident response, starting with humor about “just ask the LLM,” but landing on a real point. AI can be excellent at sifting noisy logs, even if you should not blindly trust it mid-emergency. Dave pivots to the idea that the best loyalty, from customers and coworkers, is earned when something goes wrong and support is excellent. He describes jumping into a long outage call ready to tear apart his own recent work with zero ego, because people remember who shows up with “two tow trucks” when everything’s on fire. Mike and Justin emphasize composure and delegation. If you are overwhelmed, hand off to someone with a cool head. Prioritize restoring service, “stop the bleeding,” before deep root-cause analysis. Invest ahead of time in rollback plans, feature flags, staged rollouts, and observability. From there, they broaden into practical triage and long-term resilience. Verify the issue, look at metrics and dashboards to identify symptoms like CPU, disk, network, traffic spikes, and database issues, and narrow the delta between last-known-good and broken. They discuss how constraints differ in mobile, including App Store review delays, crash loops, and reliance on the user’s device and network. They also cover security incidents, where you need monitoring to detect attacks, plus coordinated mitigation like blocking traffic and working with vendors. They stress the importance of having an incident quarterback, a playbook, and a contact list for after-hours escalation. The close focuses on what comes after the band-aid. Do postmortems and cleanup so temporary fixes do not become permanent donuts. Balance realistic risk planning with business needs. Emphasize strong observability and the ability to recover quickly, alongside prevention, echoing practices like Chaos Monkey and the idea that monitoring prevents historical events from re-happening. Transcript: MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today. We've got a good crew here today, and I'm excited about this one. We've got Kyle Archer, Eddy Lopez. We've got Dave Brady. Hello, Justin Ellis, Thomas Wilcox. We've got Ramses Bateman, and Will Archer. So, I think we've all been here before multiple times [chuckles]. We've got a familiar crew to talk about an important topic that's always fresh because [chuckles] there's a constant need. I was racking my brain what story to tell for this, and I ended up going back to...I don't even remember exactly when it was, but it was somewhere in my late teens, early twenties, in that era. So, admission, that's quite a long time ago [laughs]. That's more than halfway back [laughs]. And I was helping out at my parents' house with some remodeling they were doing. They were tearing out the...they were redoing the bathroom. And so, they were tearing out...they had a wall that had some tile on it, and they were tearing out the tile. And they were going to put some new...I don't even remember. They shifted things around, but they were tearing out the tile. That's the important part. And I had my little brother with me nearby. He was too young to really help. He was, like, six. And, you know, he was just hanging out and chatting with me, and I was taking a...they call it a dead blow hammer. It's a hammer with sand in it, so when you hit, it just stops. So, it's a weighted hammer, but it has a soft landing, so it doesn't have a...it doesn't bounce back, right? It just kind of stops, rather than having a strong bounce. It's good for situations where you want to do that, right, where you don't...you really don't want it bouncing back and hitting you in the face. And I was breaking up the tile wall. Context, there I am with, like, a six-year-old breaking up a tile wall. And there was some wire mesh behind it, and I was gradually peeling back. As I broke it, I was peeling back this wire mesh that was embedded in some sort of mortar. And I was pulling out [inaudible 02:26] the cement behind the tile. And so, as I'm banging, I pull back a piece, you know, pull it back because I'm making some progress, and I swing in. And because that broken tile is now hanging out and mounted on that wire behind, with the, you know, the cement that's holding it together, when I swing with that hammer at full force, right after peeling, you know, an extra layer back, I sunk my knuckle of my pinky finger right into a piece of broken tile. And I go, oh, and I look down. And I look down into my knuckle, maybe five eighths of an inch, a couple of centimeters more than you should be looking down into a knuckle [laughs]. Oh [laughs], that moment, that's not good. And then the blood starts, right? A rather remarkable amount of blood, I'll say [laughs], was coming out of the finger. Remember, there's a six-year-old here in the room with me. And he yells, "Mom, dad, come help Mike. He's really hurt bad." And, of course, they're thinking the worst. I'm like, "No, no, no, no, no, it's okay [laughs]," yelling. But, you know, there's the moment of panic there. And so, I had some choices in that moment, right? What do I do? Luckily, I think I handled it pretty well. I comforted the people around me to let them know this isn't a disaster. I'm going to need to do something, but you don't need to, you know, call 911. Unfortunately...so, we got everything up, went to one of those urgent care places. They stitched me up. I could tell some other weird stories about it there. A few weeks later, I noticed a little white mark on my finger, and I started pulling, and it was a piece of the thread from the gauze that had somehow got stuck in my finger. And I pulled out, like, a foot [laughs] of this string out of my finger, and then it snapped down near the bottom, and some of it zipped back in. I've never seen it again, like, oooh [vocalization] [laughs]. And I still, when I touch my knuckle, I feel weird sensations all the way down the rest of my finger. It's a [inaudible 04:21] impact of that one. But my poor little brother [chuckles], he got sick from seeing it, and he was throwing up and just not okay. And I felt bad, and I had to comfort him, "This is really okay. I get some stitches, and it'll be fine [chuckles]. It will be fine." And [chuckles] I felt really bad because I was not really even thinking about it. I didn't realize that he was not okay. So, when I discovered before I left, like, 10 minutes later, he wasn't okay, you know, I gave him a hug, you know, tried to help him feel like things were okay, get a ride over to the urgent care facility. They stitched me up, and I'm fine. Today, we're going to talk about dealing with production incidents. And I bring up this example because it's outside of software, but it's a production incident, right? You've got the bad things happen, and what do you do? What do you do now? And I think that there's some aspects to that story we can riff on as well as others. But it helps set the stage for a lot of what happens when we have these production incidents and what we do in that moment because it matters a lot. And how some of the reactions, you know, there's a variety of reactions to this moment among the various parties in place that had some better, some worse, you know, impact. So, servers are down, you know, how do you keep cool? Things are on fire. And that's our topic today. And I've got definitely some thoughts on this. I've written down some notes, but, as usual, I don't want to...I've told the story, right? I've laid out the context. So, I am really hoping some of you all will have some initial thoughts to lead out with. EDDY: Sorry, is the answer not ask AI to see what's wrong with your server [inaudible 06:02]? MIKE: [laughs] DAVE: How do you think the server went down? EDDY: I was thinking, is that not the go-to answer now? I'm sorry, podcast over. Ask the LLM. [laughter]. WILL: Not not the answer. DAVE: The AI is going to say, "You are absolutely right to be upset that the server is down." JUSTIN: So, related to that --  WILL: I mean, I'm just saying that's not not the answer. Like, AI is great at reading a log. Like, it took me --  DAVE: Yeah, actually. WILL: Years, if not decades, to get, like, pretty decent at reading log vomit, you know what I mean, like, filtering through the chicken innards that [laughter], you know, a log will, like, throw up all over you and just be like, "Oh yeah, that's actually it." AI is actually super duper at that. I don't trust it, especially in an emergency but, like, do that. Sure. Yes. Do it. EDDY: I was literally pairing with someone, and we were looking at a Grafana log, right? And I'm like, "Oh, it's because of this." And they're like, "Where? Where is that?" And I'm like, "Oh, I read it somewhere here. Hold on, let me find it again." And, like, you get so good at ignoring all the clutter, you know, and just filtering everything. But, oh my God, dude, like, AI can sift through, like, raw JSON, like candy. DAVE: I have a thought to throw out. I have a bunch. I always do. But one of the things that...and this is not really a production thing, well, maybe it is: loyalty. The thing that makes somebody loyal, a customer, in particular, is you get this graph of, like, did they have a good time, or did they have a bad time? And then did they receive good support, or did they receive bad support? And the most vehement haters of a

    1h 12m
See All (101)

Ratings & Reviews

4.5
out of 5
2 Ratings

About

At Acima, we have a large software development team. We wanted to be able to share with the community things we have learned about the development process. We'll share some tech specifics (we do Ruby, Kotlin, Javascript, and Haskell), but also talk a lot about mentoring, communication, hiring, planning, and the other things that make up a lot of the software development process but don't always get talked about enough.

Information

  • Creator
    Mike Challis
  • Years Active
    2022 - 2026
  • Episodes
    101
  • Rating
    Clean
  • Copyright
    © 2026 Acima Development
  • Show Website
    Acima Development