Adopting Airflow at Netlify The Data Flowcast: Mastering Airflow for Data Engineering & AI

After a bit of a break, we're back with the third official episode bundle of The Airflow Podcast. In this batch, we'll get a little bit deeper with current Airflow users and maintainers on core fundamental concepts in data engineering, architectures for operating modern data platforms at scale, and the process of maintaining and operating Airflow, specifically as we go through the release process of Airflow 2.0.

This week, we met up with Brian de la Motte and Florian Hines at Netlify. Netlify provides an extremely popular toolset for building and deploying JAMstack sites. They provide hosting services, CI, DNS, authentication, and managed backend tools that help users run and operate static sites at scale. The team over there recently adopted Airflow to help decouple orchestration logic from a complex collection Spark jobs and are currently in the process of expanding their Airflow footprint to accommodate a broader group of interesting use-cases.

Disclaimer: we get a bit of a surprise about halfway through the episode when Brian tells us that they had recently signed up for Astronomer- we promise that it wasn't a planted ad :).

Please contact pete@astronomer.io if you'd like to get in touch regarding future episodes. Hope you enjoy!

Guest Profiles:
Brian de la Motte: https://www.linkedin.com/in/brian-de-la-motte/
Florian Hines: https://www.linkedin.com/in/florianhines/