The Role of Python in Shaping the Future of Data Platforms with DLT

Summary
In this episode of the Data Engineering Podcast, Adrian Broderieux and Marcin Rudolph, co-founders of DLT Hub, delve into the principles guiding DLT's development, emphasizing its role as a library rather than a platform, and its integration with lakehouse architectures and AI application frameworks. The episode explores the impact of the Python ecosystem's growth on DLT, highlighting integrations with high-performance libraries and the benefits of Arrow and DuckDB. The episode concludes with a discussion on the future of DLT, including plans for a portable data lake and the importance of interoperability in data management tools.
Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• Imagine catching data issues before they snowball into bigger problems. That’s what Datafold’s new Monitors do. With automatic monitoring for cross-database data diffs, schema changes, key metrics, and custom data tests, you can catch discrepancies and anomalies in real time, right at the source. Whether it’s maintaining data integrity or preventing costly mistakes, Datafold Monitors give you the visibility and control you need to keep your entire data stack running smoothly. Want to stop issues before they hit production? Learn more at dataengineeringpodcast.com/datafold today!
• Your host is Tobias Macey and today I'm interviewing Adrian Brudaru and Marcin Rudolf, cofounders at dltHub, about the growth of dlt and the numerous ways that you can use it to address the complexities of data integration

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you describe what dlt is and how it has evolved since we last spoke (September 2023)?
  • What are the core principles that guide your work on dlt and dlthub?
• You have taken a very opinionated stance against managed extract/load services. What are the shortcomings of those platforms, and when would you argue in their favor?
• The landscape of data movement has undergone some interesting changes over the past year. Most notably, the growth of PyAirbyte and the rapid shifts around the needs of generative AI stacks (vector stores, unstructured data processing, etc.). How has that informed your product development and positioning?
  • The Python ecosystem, and in particular data-oriented Python, has also undergone substantial evolution. What are the developments in the libraries and frameworks that you have been able to benefit from?
• What are some of the notable investments that you have made in the developer experience for building dlt pipelines?
  • How have the interfaces for source/destination development improved?
• You recently published a post about the idea of a portable data lake. What are the missing pieces that would make that possible, and what are the developments/technologies that put that idea within reach?
• What is your strategy for building a sustainable product on top of dlt?
  • How does that strategy help to form a "virtuous cycle" of improving the open source foundation?
• What are the most interesting, innovative, or unexpected ways that you have seen dlt used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on dlt?
• When is dlt the wrong choice?
• What do you have planned for the future of dlt/dlthub?

Contact Info

• Adrian
  • LinkedIn
• Marcin
  • LinkedIn

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data ma