Data Engineering Central Podcast

Data Engineering in Real Life

Long Live the Data Engineer. No holds barred. Talking about Data Engineering news, topics, and general mayhem. dataengineeringcentral.substack.com

  1. 10 giờ trước

    The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering

    The creator of Pandas and co-creator of Apache Arrow, Wes McKinney, joins the Data Engineering Central Podcast for an in-depth conversation about how modern data engineering came to exist, where AI is taking software development, and why good engineering still matters more than ever. We start with Wes’ journey from building GoldenEye fan websites as a teenager to creating Pandas while working at a quantitative hedge fund, and eventually launching Apache Arrow, one of the foundational technologies behind today’s modern data ecosystem. Along the way, we discuss Cloudera, Parquet, DuckDB, DataFusion, Spark, and how the industry evolved from Hadoop to today’s lakehouse architectures. Thanks for reading Data Engineering Central! This post is public so feel free to share it. The second half of the conversation dives deep into AI. Wes explains why large language models make experienced engineers more productive but won’t magically replace software engineering, why architecture and good taste are becoming more valuable than writing individual lines of code, and why projects like DuckDB and * Apache Arrow remains incredibly difficult to recreate with AI alone. We also discuss open-source, local AI models, token costs, multimodal data platforms, and what new engineers should focus on to build long-term careers in software and data. If you’re a data engineer, software engineer, architect, engineering leader, or simply interested in where AI is taking our industry, this is a conversation you won’t want to miss. Topics We Cover * How Pandas was created * The story behind Apache Arrow * Why Arrow became the standard for modern data systems * DuckDB, DataFusion, and the next generation of data tools * The evolution from Hadoop to lakehouses * Why AI won’t replace great software engineers * Architecture vs. coding in the AI era * Building trustworthy open source software * The future of data engineering * Advice for new engineers entering the industry If you enjoy conversations with the people building the future of data engineering, subscribe for more interviews with the creators of the tools we use every day. Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    1 giờ
  2. From COBOL to Copilot: 30 Years of Data, BI, and AI with David Langer

    1 thg 7

    From COBOL to Copilot: 30 Years of Data, BI, and AI with David Langer

    What happens when someone who started programming on a Commodore 64 watches AI reshape the entire data industry? In this episode of the Data Engineering Central Podcast, I sit down with Dave Langer to explore nearly three decades of experience across software engineering, business intelligence, analytics, data science, and AI. Dave’s career spans COBOL programming on IBM mainframes, enterprise architecture, Microsoft’s Xbox division, machine learning, startup leadership, authorship, and building one of the largest personal brands in the data space. We discuss why many of the biggest problems in data haven’t changed, even as the tools continue to evolve. We dive into the reality behind self-service analytics, the importance of dimensional modeling, what organizations are getting wrong about AI adoption, and why developing strong analytical skills matters more than ever. * Dave also shares practical advice for data professionals navigating the AI era, explaining why tools like Copilot should be viewed as partners rather than replacements. If you’re a data analyst, BI developer, data scientist, or data engineer wondering what the future holds, this conversation offers both perspective and optimism. Thanks for reading Data Engineering Central! This post is public so feel free to share it. What We Cover * How Dave got started programming on a Commodore 64 * The transition from COBOL to modern analytics * Why the core problems in data haven’t changed * The evolution of business intelligence and dashboards * How Dave discovered machine learning * Why data science needs more than just Jupyter notebooks * The limitations of self-service analytics * Why semantic layers and governance matter for AI * Advice for staying relevant as AI reshapes the industry * Building a personal brand in data * Writing a technical book and becoming an independent creator Connect with Dave: * LinkedIn: https://www.linkedin.com/in/davelanger/ * Substack: The DIY Data Scientist * Book: Python and Excel Step-by-Step Subscribe for more conversations on data engineering, analytics, AI, and building a career in modern data. Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    57 phút
  3. Semantic Layers, Agents, and the Future of Analytics

    24 thg 6

    Semantic Layers, Agents, and the Future of Analytics

    In this episode of the Data Engineering Central Podcast, I sit down with David Jaitillake to explore the future of data engineering, analytics, and AI. David has spent nearly two decades working across data teams, from analyst roles in the early SQL Server days to leading teams, founding startups, serving as VP of AI at Cube, and now co-founding Quarry. We discuss why semantic layers have suddenly become one of the most important concepts in modern data platforms, how tools like Claude Code are transforming engineering workflows, and why the core problems in data haven’t really changed despite massive advances in technology. David shares his perspective on where agentic workflows are headed, what AI means for junior engineers entering the field, and why experienced practitioners may be more valuable than ever before. We also dive into the evolution of data platforms, lessons learned from startups, the promise of tools like DuckDB and MotherDuck, and how organizations should think about adopting AI responsibly. Thanks for reading Data Engineering Central! This post is public so feel free to share it. Whether you’re a data engineer, analytics engineer, engineering leader, or someone trying to understand where the industry is headed, this conversation offers a practical and honest look at what’s coming next. What We Cover * David’s journey from analyst to startup founder * The rise of semantic layers and why they matter * Why data modeling is still critical in the AI era * How AI coding agents are changing engineering work * What Claude Code is enabling today * The future of agentic data pipelines * Why DuckDB and MotherDuck are gaining traction * The challenges facing junior engineers * Career advice for data professionals at every stage * Whether David is optimistic about the future of AI and data Connect with David: * LinkedIn: https://www.linkedin.com/in/david-jayatillake/ * Substack: Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    45 phút
  4. The Future of the Lakehouse: Delta Lake, Rust, and Data Platforms at Scale

    17 thg 6

    The Future of the Lakehouse: Delta Lake, Rust, and Data Platforms at Scale

    In this episode of the Data Engineering Central Podcast, I sit down with Ethan, a maintainer of delta-rs and an expert in modern lakehouse architecture working in the pharmaceutical industry. We discuss Ethan’s journey into tech and data engineering, the evolution of open table formats like Delta Lake and Apache Iceberg, and what it actually takes to build scalable enterprise data platforms in highly regulated environments like big pharma. We also dive into: * delta-rs and the future of Delta Lake outside Spark * Lakehouse architecture and open catalogs * Rust in the modern data ecosystem * Data platform governance and scalability * Enterprise analytics and infrastructure * The future of agentic analytics and AI-enabled data systems * Lessons learned building large-scale data platforms If you’re interested in modern data engineering, open source infrastructure, lakehouses, or the future of analytics engineering, this is a great conversation. Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    56 phút
  5. From Failure to AWS: What Actually Makes a Great Engineer

    10 thg 6

    From Failure to AWS: What Actually Makes a Great Engineer

    Victor Moreno went from failing out of a top CS program to becoming a senior engineer at AWS, and his story says a lot about what actually matters in software engineering today. In this conversation, we go deep into the reality behind the AI hype, what makes engineers valuable (it’s not writing more code), and why the future of the field looks very different from what most people think. We talk about the shift from coding to system thinking, why fundamentals matter more in the age of AI, and how junior engineers will need to evolve as tools like Claude and ChatGPT take over the “grunt work.” Victor also shares hard-earned lessons from teaching, startups, consulting, and building systems at AWS, along with practical advice for engineers looking to stand out in a crowded, uncertain job market. This is not a hype conversation. It’s a real look at where things are going. Thanks for reading Data Engineering Central! This post is public so feel free to share it. 🔑 What We Cover * Why AI is making fundamentals more important, not less * The biggest mistake engineers make is chasing promotions * How to actually become a high-impact engineer * Why does doing more Jira tickets not matter * What’s broken about today’s interview process * The future of junior engineers in an AI world * Tactical vs strategic engineering (and why it matters) * Why most AI-generated code is still “low quality.” * How to think about career growth in a weird job market 💡 Key Takeaway The best engineers aren’t the ones writing the most code—they’re the ones who understand systems, think long-term, and can drive decisions. Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    52 phút
  6. How Real Data Engineers Think (Beyond Tools and Hype)

    3 thg 6

    How Real Data Engineers Think (Beyond Tools and Hype)

    In this episode of the Data Engineering Central Podcast, I sit down with Yordan Ivanov, Head of Data Engineering at a growing fintech company, to talk through what it actually looks like to build and run real data platforms in production. Yordan’s story starts like many of mine, early programming, gaming, PHP, Linux servers—but what makes this conversation interesting is how he evolved from a generalist software engineer into a data engineering leader without even realizing it at first. We spend a lot of time digging into what actually matters in modern data engineering, and it’s not the tools. Thanks for reading Data Engineering Central! This post is public so feel free to share it. We talk about: * Why the industry went too far into complexity and is now swinging back toward simplicity * The reality of running a data platform at scale (and why most teams waste time maintaining tools instead of delivering value) * How to think about migrations the right way without breaking everything * The difference between junior, mid, and senior engineers—and why ambiguity tolerance and impact matter more than coding ability * Why “perfect” engineering is a trap and how to actually ship work that matters We also get into AI, and Yordan has one of the more grounded takes you’ll hear right now. Most companies aren’t even close to ready for AI, and the idea that it’s replacing engineers anytime soon misses the bigger problem: messy data, unclear metrics, and weak foundations. Check out Yordan’s Substack below! We also talk about: * How AI is actually used on real teams today (not Twitter hype) * Why juniors with AI can be risky without strong processes * How to think about code reviews, testing, and slowing down when it matters On top of that, we dig into content creation, Substack, and what it takes to stand out in a world full of generic AI-generated content. Yordan’s approach is simple: write from real experience or don’t write at all. This is one of those conversations that cuts through a lot of noise and gets back to fundamentals, how to think, how to build, and how to grow as an engineer in a rapidly changing space. Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    49 phút
  7. Data, AI, and DuckDB

    27 thg 5

    Data, AI, and DuckDB

    In this episode of the Data Engineering Central Podcast, I sit down with Jacob Matson, Developer Advocate at MotherDuck, to unpack one of the most interesting shifts happening in data engineering right now. Jacob didn’t start in tech the way most people expect. He began in accounting, working with Excel and financial systems, before slowly realizing that the real problem he loved solving wasn’t finance, it was data pipelines. That path eventually led him deep into SQL Server, data warehousing, and ultimately to DuckDB, a tool that fundamentally changed how he thought about processing data. * What we get into is bigger than just tools, though. We talk about why DuckDB exploded in popularity, what it gets right that traditional databases and even modern cloud warehouses struggle with, and why the industry may be swinging back toward simplicity after years of over-engineered “modern data stacks.” There’s a really interesting thread here around how engineers accidentally created too much complexity, and now tools like DuckDB are winning by removing it. We also go deep on the evolution of the data stack itself. From SQL Server’s “everything in one box” model, to the unbundled chaos of the modern stack, and now back toward a more unified, simpler approach. Jacob shares how MotherDuck is thinking about that shift and where things are headed next. * One of the more important parts of this conversation is around AI. There’s a strong argument here that AI doesn’t kill data engineering; it massively expands it. Instead of fewer queries being written, we may be heading toward a world where AI agents generate orders of magnitude more queries than humans ever could. That flips a lot of assumptions on their head, especially around things like data modeling, which suddenly becomes more important, not less. We also talk about: * Why most Spark workloads are overkill * When single-node tools like DuckDB actually win * The real tradeoffs behind Lakehouse architectures * Why data modeling is still critical in an AI-driven world * How engineers should think about building in 2026 and beyond This is one of those conversations that helps you zoom out and see where things are actually going, not just what tools are trending this week. If you’re building data platforms, experimenting with AI, or just trying to simplify your stack, this one is worth your time. Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    50 phút
  8. Why I Left Facebook to Work for Myself

    20 thg 5

    Why I Left Facebook to Work for Myself

    In this episode of the Data Engineering Central Podcast, I sit down with Ben Rogojan to talk about the real story behind data engineering careers, Big Tech, and what’s changing right now. Ben shares how he went from working in kitchens… to data engineering… to Facebook… and eventually walking away from it all to build his own consulting business. And yeah, it wasn’t all glamorous. “I was making the same money as Facebook… and I hated my life.” We get into the stuff most people don’t talk about: * What it’s actually like working in Big Tech * Why high-paying jobs can still burn you out * How he transitioned into consulting (and what people get wrong) * The reality of modern data stacks and tool sprawl * Whether data engineering is changing because of AI * Why fundamentals still matter more than ever Thanks for reading Data Engineering Central! This post is public so feel free to share it. We also go deep on where the industry is heading: * Is the “modern data stack” breaking down? * Are tools like DuckDB actually replacing warehouses? * Is data modeling dead… or just not trendy anymore? * What AI is really changing (and what it’s not) If you’re trying to break into data, grow your career, or figure out where things are headed… this is one of the more honest conversations you’ll hear. Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. Ben also runs a course and community for those interested in getting into consulting. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

    53 phút
Xem tất cả (36)

Giới Thiệu

Long Live the Data Engineer. No holds barred. Talking about Data Engineering news, topics, and general mayhem. dataengineeringcentral.substack.com

Thông Tin

  • Nhà sáng tạo
    Data Engineering in Real Life
  • Năm hoạt động
    2024 - 2026
  • Tập
    36
  • Xếp hạng
    Thô tục
  • Bản quyền
    © dataengineeringdude
  • Trang web chương trình
    Data Engineering Central Podcast

Có Thể Bạn Cũng Thích