Data Engineering Central Podcast

Data Engineering in Real Life

Long Live the Data Engineer. No holds barred. Talking about Data Engineering news, topics, and general mayhem. dataengineeringcentral.substack.com

  1. 2d ago

    Data, AI, and DuckDB

    In this episode of the Data Engineering Central Podcast, I sit down with Jacob Matson, Developer Advocate at MotherDuck, to unpack one of the most interesting shifts happening in data engineering right now. Jacob didn’t start in tech the way most people expect. He began in accounting, working with Excel and financial systems, before slowly realizing that the real problem he loved solving wasn’t finance, it was data pipelines. That path eventually led him deep into SQL Server, data warehousing, and ultimately to DuckDB, a tool that fundamentally changed how he thought about processing data. * What we get into is bigger than just tools, though. We talk about why DuckDB exploded in popularity, what it gets right that traditional databases and even modern cloud warehouses struggle with, and why the industry may be swinging back toward simplicity after years of over-engineered “modern data stacks.” There’s a really interesting thread here around how engineers accidentally created too much complexity, and now tools like DuckDB are winning by removing it. We also go deep on the evolution of the data stack itself. From SQL Server’s “everything in one box” model, to the unbundled chaos of the modern stack, and now back toward a more unified, simpler approach. Jacob shares how MotherDuck is thinking about that shift and where things are headed next. * One of the more important parts of this conversation is around AI. There’s a strong argument here that AI doesn’t kill data engineering; it massively expands it. Instead of fewer queries being written, we may be heading toward a world where AI agents generate orders of magnitude more queries than humans ever could. That flips a lot of assumptions on their head, especially around things like data modeling, which suddenly becomes more important, not less. We also talk about: * Why most Spark workloads are overkill * When single-node tools like DuckDB actually win * The real tradeoffs behind Lakehouse architectures * Why data modeling is still critical in an AI-driven world * How engineers should think about building in 2026 and beyond This is one of those conversations that helps you zoom out and see where things are actually going, not just what tools are trending this week. If you’re building data platforms, experimenting with AI, or just trying to simplify your stack, this one is worth your time. Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    50 min

  2. May 20

    Why I Left Facebook to Work for Myself

    In this episode of the Data Engineering Central Podcast, I sit down with Ben Rogojan to talk about the real story behind data engineering careers, Big Tech, and what’s changing right now. Ben shares how he went from working in kitchens… to data engineering… to Facebook… and eventually walking away from it all to build his own consulting business. And yeah, it wasn’t all glamorous. “I was making the same money as Facebook… and I hated my life.” We get into the stuff most people don’t talk about: * What it’s actually like working in Big Tech * Why high-paying jobs can still burn you out * How he transitioned into consulting (and what people get wrong) * The reality of modern data stacks and tool sprawl * Whether data engineering is changing because of AI * Why fundamentals still matter more than ever Thanks for reading Data Engineering Central! This post is public so feel free to share it. We also go deep on where the industry is heading: * Is the “modern data stack” breaking down? * Are tools like DuckDB actually replacing warehouses? * Is data modeling dead… or just not trendy anymore? * What AI is really changing (and what it’s not) If you’re trying to break into data, grow your career, or figure out where things are headed… this is one of the more honest conversations you’ll hear. Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. Ben also runs a course and community for those interested in getting into consulting. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    53 min

  3. May 13

    Academic → CTO: What Actually Matters in Data (Matthew Housley)

    Most companies don’t have a tooling problem. They have a foundation problem. In this episode, I sit down with Matthew Housley, a famed co-author of Data Engineering Fundamentals and former CTO of Ternary Data, to talk about what actually makes data teams successful and why so many organizations get it wrong despite having modern stacks, cloud platforms, and expensive dashboards. * Matthew’s path is a little different than most. He started in academia as a mathematics instructor before moving into industry as a data scientist at Overstock.com, and eventually leading data strategy and analytics as a CTO. That mix of academic rigor and real-world execution gives him a very clear perspective on where things break down. We get into the gap between data science and real business impact, why analytics foundations matter more than flashy models, and what companies consistently underestimate when building out data platforms. We also talk about what it actually looks like to transition from academia to industry, and how that shapes how you think about data problems at scale. Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. If you’ve ever felt like your data stack should be delivering more value than it is, this conversation will probably hit close to home. ⏱️ Topics we cover: * Why most analytics efforts fail before they even start * The difference between “doing data” and delivering value * Data science vs data engineering vs analytics reality * Academic thinking vs industry execution * What CTOs actually care about when it comes to data * Building foundations that don’t fall apart six months later Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    56 min

  4. May 6

    AI Isn’t Replacing Curious Developers

    AI isn’t just changing how we write code. It’s changing what it even means to build software. In this episode of the Data Engineering Central Podcast, I sit down with Neil Roberts — a developer who’s been through every major wave of the web, from BASIC on an Atari to modern TypeScript, and now deep into LLMs and agentic workflows. This is not another surface-level “AI will change everything” conversation. We get into what is actually happening right now, where it works, where it completely breaks, and what developers are getting wrong. * We talk about why front-end and UX matter more than ever in an AI world, why most people misunderstand agents, and what real day-to-day workflows with LLMs actually look like. * There’s also a hard look at who benefits from AI, who falls behind, and whether we are quietly building fragile systems that we don’t fully understand. If you’re a developer trying to figure out where this is all going, this is one of those conversations worth paying attention to. Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. Expect to learn: * Why AI is as much a UX problem as it is a backend problem * What “agents” actually mean in practice, not in demos * Where LLM workflows are useful today and where they fail hard * Whether junior developers should be worried or excited * How building apps changes when AI is part of the system * What developers should actually be doing right now to stay relevant Neil also has a podcast, The Skill Tree, on AI and agentic-specific topics. We also get into a bigger question most people are avoiding: * Are we heading toward AI-assisted coding… or AI-orchestrated systems where developers become supervisors? * And maybe more importantly… which side of that shift do you want to be on? Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    1h 4m

  5. Apr 29

    AI Is Changing Data Engineering Fast

    In this episode of the Data Engineering Central Podcast, I sit down with Andreas Kretz to break down what is really happening in the industry right now. We go far beyond surface-level AI hype and talk about how data engineering actually works in the real world, what skills still matter, and where most engineers are wasting time. Andreas shares his full journey from industrial IoT and working at Bosch to building one of the largest data engineering education platforms in the world, training over 2,000 students and reaching more than 100,000 engineers globally. We get into what production data systems actually look like, why most learning paths are broken, and how AI is reshaping the role of the modern data engineer. Thanks for reading Data Engineering Central! This post is public so feel free to share it. * We also dig into the uncomfortable truths. AI can write code, but it cannot replace thinking. Most engineers focus too much on tools and not enough on problem-solving, system design, and communication. That gap is only getting bigger. If you are trying to figure out how to stay relevant in data engineering, or you are just getting started and want to avoid years of wasted effort, this conversation will change how you think about your career. Today’s podcast is sponsored by Estuary. Without them, content like this isn’t possible. The best way to support this Newsletter is to check out what Estuary has to offer and click the links below. Build millisecond-latency, scalable, future-proof data pipelines in minutes. Estuary is the Right-Time Data Platform that integrates all of the systems you use to produce, process, and consume data. Also, providing best-in-class CDC (Change Data Capture). Estuary unifies today’s batch and streaming paradigms so that your systems, current and future, are synchronized around the same datasets, updating in milliseconds. What we cover: * Why most data engineers are learning the wrong things * The shift from coding to problem-solving and system design * How AI is actually changing data engineering workflows * Why courses and tutorials are becoming less effective * The difference between real production systems and “toy projects.” * The future of data engineering jobs and whether AI will replace them * Why fundamentals still matter more than ever One of the biggest takeaways is simple. The tools will keep changing, but the problems stay the same. The engineers who win are those who understand systems, ask better questions, and connect business problems to real solutions. Links: * Learn Data Engineering Academy: https://learndataengineering.com * Andreas Kretz on LinkedIn * Andreas Kretz on YouTube * Sponsor: https://estuary.dev Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    57 min
  6. Most Data Teams Are Doing It Wrong

    Apr 22

    Most Data Teams Are Doing It Wrong

    Most data teams think they’re building value. In reality, they’ve become ticket queues. In this episode, Chris Gambill explains his storied career in tech and data through the years, dealing with data at Fortune 500 company scale, and breaking out on his own. We cover career growth, what separates senior engineers from true strategic operators, and the biggest mistakes people make early on. We discuss the classic problems that have plagued data teams for decades and why it’s all still a struggle. Today’s podcast is sponsored by Estuary. Without them, content like this isn’t possible. The best way to support this Newsletter is to check out what Estuary has to offer and click the links below. Build millisecond-latency, scalable, future-proof data pipelines in minutes. Estuary is the Right-Time Data Platform that integrates all of the systems you use to produce, process, and consume data. Also, providing best-in-class CDC (Change Data Capture). Estuary unifies today’s batch and streaming paradigms so that your systems, current and future, are synchronized around the same datasets, updating in milliseconds. We also dig into Databricks vs Snowflake, what matters and what doesn’t, and how to think about modern data architecture without falling for marketing hype. * On the AI side, we talk about why most LLMs, in the context of developer lifecycles, have changed how we do data, and also about what human skills cannot be replaced. If you care about leveling up beyond just building pipelines, this one is for you. Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    59 min

  7. Apr 15

    From Industrial Data at BASF to Delta Lake Committer

    In this episode, Robert Pack walks through his journey from engineering and simulation work to building large-scale data systems across 900+ plants at BASF. We break down what those systems actually looked like, including ingestion, modeling, and the realities of batch vs real-time in industrial environments. We also dive into: * AI Workflows for Developers * His work as a committer on Delta Lake * Where lakehouse architecture works and where it falls short * The transition into Developer Relations at Databricks This is a grounded, practical conversation about what actually matters when building data platforms. Today’s podcast is sponsored by Estuary. Without them, content like this isn’t possible. The best way to support this Newsletter is to check out what Estuary has to offer and click the links below. Build millisecond-latency, scalable, future-proof data pipelines in minutes. Estuary is the Right-Time Data Platform that integrates all of the systems you use to produce, process, and consume data. Also, providing best-in-class CDC (Change Data Capture). Estuary unifies today’s batch and streaming paradigms so that your systems, current and future, are synchronized around the same datasets, updating in milliseconds. You can find Robert on LinkedIn and GitHub, below. Thanks for reading Data Engineering Central! This post is public so feel free to share it. Come follow me on YouTube!! This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    48 min

  8. Apr 1

    He Quit Apple After 13 Years

    In this episode of Data Engineering Central, I sit down with Kevin, who spent 13 years working at Apple before walking away at the end of 2025. * Not to jump to another job. * Not to start a company. * But to take a step back from everything. Kevin shares his full journey—from growing up in the suburbs of Atlanta to building a career at Apple, and ultimately reaching the point where he could walk away financially and mentally. You can follow along with Kevin below. We dive deep into what it’s really like working in tech: the high salaries, the lifestyle creep, the pressure, and the surprising reality that even people making great money often have no clear financial plan. This conversation also explores the rise of FIRE (Financial Independence, Retire Early), how Kevin discovered it through Mr. Money Mustache, and why his perspective on it has changed over time. Thanks for reading Data Engineering Central! This post is public so feel free to share it. What starts as a path to freedom can easily turn into a scarcity mindset—and that’s something most people don’t talk about. We also get into: * Why high income does not equal financial freedom * The hidden trap of lifestyle inflation in tech * The simple investing strategy that actually works (and why most people ignore it) * Why many engineers are “close” to freedom—but never pull the trigger * The psychology of money, status, and why people stay stuck * How a failed project and burnout became a turning point * And how Kevin went from overworked and unhealthy… to climbing mountains and preparing to backpack 1,000 miles This is not your typical “get rich quick” or “retire at 30” conversation. It’s a grounded, honest look at money, work, and what it actually takes to build a life you don’t need to escape from. If you work in tech, think about FIRE, or just feel like you’re stuck on the treadmill, this one will hit home. Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    52 min
See All (30)

About

Long Live the Data Engineer. No holds barred. Talking about Data Engineering news, topics, and general mayhem. dataengineeringcentral.substack.com

Information

  • Creator
    Data Engineering in Real Life
  • Years Active
    2024 - 2026
  • Episodes
    30
  • Rating
    Explicit
  • Copyright
    © dataengineeringdude
  • Show Website
    Data Engineering Central Podcast

You Might Also Like