The Data Flowcast: Mastering Airflow for Data Engineering & AI

Astronomer
  • 5.0 (20)
  • TECHNOLOGY
  • UPDATED BIWEEKLY
The Data Flowcast: Mastering Airflow for Data Engineering & AI

Welcome to The Data Flowcast: Mastering Airflow for Data Engineering & AI — the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward. Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. Podcast Webpage: https://www.astronomer.io/podcast/

  1. Exploring the Power of Airflow 3 at Astronomer with Amogh Desai

    DEC 20

    Exploring the Power of Airflow 3 at Astronomer with Amogh Desai

    What does it take to go from fixing a broken link to becoming a committer for one of the world’s leading open-source projects?  Amogh Desai, Senior Software Engineer at Astronomer, takes us through his journey with Apache Airflow. From small contributions to building meaningful connections in the open-source community, Amogh’s story provides actionable insights for anyone on the cusp of their open-source journey. Key Takeaways: (02:09) Building data engineering platforms at Cloudera with Kubernetes. (04:00) Brainstorming led to contributing to Apache Airflow. (05:17) Starting small with link fixes, progressing to Breeze development. (07:00) Becoming a committer for Apache Airflow in September 2023. (09:51) The steep learning curve for contributing to Airflow. (16:30) Using GitHub’s “good-first-issue” label to get started. (18:15) Setting up a development environment with Breeze. (22:00) Open-source contributions enhance your resume and career. (24:51) Amogh’s advice: Start small and stay consistent. (28:12) Engage with the community via Slack, email lists and meetups. Resources Mentioned: Amogh Desai - https://www.linkedin.com/in/amogh-desai-385141157/?originalSubdomain=in%20%20https://www.linkedin.com/company/astronomer/ Astronomer - https://www.linkedin.com/company/astronomer/ Apache Airflow GitHub Repository - https://github.com/apache/airflow Contributors Quick Guide - https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst Breeze Development Tool - https://github.com/apache/airflow/tree/main/dev/breeze Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

    30 min
  2. Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta

    DEC 12

    Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta

    Data orchestration and machine learning are shaping how organizations handle massive datasets and drive customer-focused strategies. Tools like Apache Airflow are central to this transformation. In this episode, Vasyl Vasyuta, R&D Team Leader at Optimove, joins us to discuss how his team leverages Airflow to optimize data processing, orchestrate machine learning models and create personalized customer experiences. Key Takeaways:  (01:59) Optimove tailors marketing notifications with personalized customer journeys. (04:25) Airflow orchestrates Snowflake procedures for massive datasets. (05:11) DAGs manage workflows with branching and replay plugins. (05:41) The "Joystick" plugin enables seamless data replays. (09:33) Airflow supports MLOps for customer data grouping. (11:15) Machine learning predicts customer behavior for better campaigns. (13:20) Thousands of DAGs run every five minutes for data processing. (15:36) Custom versioning allows rollbacks and gradual rollouts. (18:00) Airflow logs enhance operational observability. (23:00) DAG versioning in Airflow 3.0 could boost efficiency. Resources Mentioned: Vasyl Vasyuta - https://www.linkedin.com/in/vasyl-vasyuta-3270b54a/ Optimove - https://www.linkedin.com/company/optimove/ Apache Airflow - https://airflow.apache.org/ Snowflake - https://www.snowflake.com/ Datadog - https://www.datadoghq.com/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

    24 min
  3. Maximizing Business Impact Through Data at GlossGenius with Katie Bauer

    DEC 5

    Maximizing Business Impact Through Data at GlossGenius with Katie Bauer

    Bridging the gap between data teams and business priorities is essential for maximizing impact and building value-driven workflows. Katie Bauer, Senior Director of Data at GlossGenius, joins us to share her principles for creating effective, aligned data teams. In this episode, Katie draws from her experience at GlossGenius, Reddit and Twitter to highlight the common pitfalls data teams face and how to overcome them. She offers practical strategies for aligning team efforts with organizational goals and fostering collaboration with stakeholders.  Key Takeaways: (02:36) GlossGenius provides an all-in-one platform for beauty professionals. (03:59) Airflow orchestrates data and MLOps workflows at GlossGenius. (04:41) Focusing on value helps data teams achieve greater impact. (06:23) Aligning team priorities with company goals minimizes friction. (08:44) Building strong stakeholder relationships requires curiosity. (12:46) Treating roles as flexible fosters team innovation. (13:21) Adapting to new technologies improves effectiveness. (18:28) Acting like your time is valuable earns respect. (23:38) Proactive data initiatives drive strategic value. (24:20) Usage data offers critical insights into tool effectiveness. Resources Mentioned: Katie Bauer - https://www.linkedin.com/in/mkatiebauer/ GlossGenius - https://www.linkedin.com/company/glossgenius/ Apache Airflow - https://airflow.apache.org/ DBT - https://www.getdbt.com/ Cosmos - https://cosmos.apache.org/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

    26 min
  4. Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade

    DEC 2

    Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade

    Scaling deployments for a billion users demands innovation, precision and resilience. In this episode, we dive into how LinkedIn optimizes its continuous deployment process using Apache Airflow. Rahul Gade, Staff Software Engineer at LinkedIn, shares his insights on building scalable systems and democratizing deployments for over 10,000 engineers.  Rahul discusses the challenges of managing large-scale deployments across 6,000 services and how his team leverages Airflow to enhance efficiency, reliability and user accessibility. Key Takeaways: (01:36) LinkedIn minimizes human involvement in production to reduce errors. (02:00) Airflow powers LinkedIn’s Continuous Deployment platform. (05:43) Continuous deployment adoption grew from 8% to a targeted 80%. (11:25) Kubernetes ensures scalability and flexibility for deployments. (12:04) A custom UI offers real-time deployment transparency. (16:23) No-code YAML workflows simplify deployment tasks. (17:18) Canaries and metrics ensure safe deployments across fabrics. (20:45) A gateway service ensures redundancy across Airflow clusters. (24:22) Abstractions let engineers focus on development, not logistics. (25:20) Multi-language support in Airflow 3.0 simplifies adoption. Resources Mentioned: Rahul Gade - https://www.linkedin.com/in/rahul-gade-68666818/ LinkedIn - https://www.linkedin.com/company/linkedin/ Apache Airflow - https://airflow.apache.org/ Kubernetes - https://kubernetes.io/ Open Policy Agent (OPA) - https://www.openpolicyagent.org/ Backstage - https://backstage.io/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

    28 min
  5. How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari

    NOV 14

    How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari

    When data orchestration reaches Uber’s scale, innovation becomes a necessity, not a luxury. In this episode, we discuss the innovations behind Uber’s unique Airflow setup. With our guests Shobhit Shah and Sumit Maheshwari, both Staff Software Engineers at Uber, we explore how their team manages one of the largest data workflow systems in the world. Shobhit and Sumit walk us through the evolution of Uber’s Airflow implementation, detailing the custom solutions that support 200,000 daily pipelines. They discuss Uber's approach to tackling complex challenges in data orchestration, disaster recovery and scaling to meet the company’s extensive data needs. Key Takeaways: (02:03) Airflow as a service streamlines Uber’s data workflows. (06:16) Serialization boosts security and reduces errors. (10:05) Java-based scheduler improves system reliability. (13:40) Custom recovery model supports emergency pipeline switching. (15:58) No-code UI allows easy pipeline creation for non-coders. (18:12) Backfill feature enables historical data processing. (22:06) Regular updates keep Uber aligned with Airflow advancements. (26:07) Plans to leverage Airflow’s latest features. Resources Mentioned: Shobhit Shah - https://www.linkedin.com/in/shahshobhit/ Sumit Maheshwar - https://www.linkedin.com/in/maheshwarisumit/ Uber - https://www.linkedin.com/company/uber-com/ Apache Airflow - https://airflow.apache.org/ Airflow Summit - https://airflowsummit.org/ Uber - https://www.uber.com/tw/en/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

    29 min
  6. Building Resilient Data Systems for Modern Enterprises at Astrafy with Andrea Bombino

    NOV 7

    Building Resilient Data Systems for Modern Enterprises at Astrafy with Andrea Bombino

    Efficient data orchestration is the backbone of modern analytics and AI-driven workflows. Without the right tools, even the best data can fall short of its potential. In this episode, Andrea Bombino, Co-Founder and Head of Analytics Engineering at Astrafy, shares insights into his team’s approach to optimizing data transformation and orchestration using tools like datasets and Pub/Sub to drive real-time processing. Andrea explains how they leverage Apache Airflow and Google Cloud to power dynamic data workflows.  Key Takeaways: (01:55) Astrafy helps companies manage data using Google Cloud. (04:36) Airflow is central to Astrafy’s data engineering efforts. (07:17) Datasets and Pub/Sub are used for real-time workflows. (09:59) Pub/Sub links multiple Airflow environments. (12:40) Datasets eliminate the need for constant monitoring. (15:22) Airflow updates have improved large-scale data operations. (18:03) New Airflow API features make dataset updates easier. (20:45) Real-time orchestration speeds up data processing for clients. (23:26) Pub/Sub enhances flexibility across cloud environments. (26:08) Future Airflow features will offer more control over data workflows. Resources Mentioned: Andrea Bombino - https://www.linkedin.com/in/andrea-bombino/ Astrafy - https://www.linkedin.com/company/astrafy/ Apache Airflow - https://airflow.apache.org/ Google Cloud - https://cloud.google.com/ dbt - https://www.getdbt.com/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

    28 min
  7. Inside Airflow 3: Redefining Data Engineering with Vikram Koka

    OCT 31

    Inside Airflow 3: Redefining Data Engineering with Vikram Koka

    Data orchestration is evolving faster than ever and Apache Airflow 3 is set to revolutionize how enterprises handle complex workflows. In this episode, we dive into the exciting advancements with Vikram Koka, Chief Strategy Officer at Astronomer and PMC Member at The Apache Software Foundation. Vikram shares his insights on the evolution of Airflow and its pivotal role in shaping modern data-driven workflows, particularly with the upcoming release of Airflow 3. Key Takeaways: (02:36) Vikram leads Astronomer’s engineering and open-source teams for Airflow. (05:26) Airflow enables reliable data ingestion and curation. (08:17) Enterprises use Airflow for mission-critical data pipelines. (11:08) Airflow 3 introduces major architectural updates. (13:58) Multi-cloud and edge deployments are supported in Airflow 3. (16:49) Event-driven scheduling makes Airflow more dynamic. (19:40) Tasks in Airflow 3 can run in any language. (22:30) Multilingual task support is crucial for enterprises. (25:21) Data assets and event-based integration enhance orchestration. (28:12) Community feedback plays a vital role in Airflow 3. Resources Mentioned: Vikram Koka - https://www.linkedin.com/in/vikramkoka/ Astronomer - https://www.linkedin.com/company/astronomer/ The Apache Software Foundation LinkedIn - https://www.linkedin.com/company/the-apache-software-foundation/ Apache Airflow LinkedIn - https://www.linkedin.com/company/apache-airflow/ Apache Airflow - https://airflow.apache.org/ Astronomer - https://www.astronomer.io/ The Apache Software Foundation - https://www.apache.org/ Join the Airflow slack and/or Dev list - https://airflow.apache.org/community/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

    30 min
  8. Building a Data-Driven HR Platform at 15Five with Guy Dassa

    OCT 24

    Building a Data-Driven HR Platform at 15Five with Guy Dassa

    Data and AI are revolutionizing HR, empowering leaders to measure performance and drive strategic decisions like never before.  In this episode, we explore the transformation of HR technology with Guy Dassa, Chief Technology Officer at 15Five, as he shares insights into their evolving data platform. Guy discusses how 15Five equips HR leaders with tools to measure and take action on team performance, engagement and retention. He explains their data-driven approach, highlighting how Apache Airflow supports their data ingestion, transformation, and AI integration. Key Takeaways: (01:54) 15Five acts as a command center for HR leaders. (03:40) Tools like performance reviews, engagement surveys, and an insights dashboard guide actionable HR steps. (05:33) Data visualization, insights, and action recommendations enhance HR effectiveness to improve their people's outcomes. (07:08) Strict data confidentiality and sanitized AI model training. (09:21) Airflow is central to data transformation and enrichment. (11:15) Airflow enrichment DAGs integrate AI models. (13:33) Integration of Airflow and DBT enables efficient data transformation. (15:28) Synchronization challenges arise with reverse ETL processes. (17:10) Future plans include deeper Airflow integration with AI. (19:31) Emphasizing the need for DAG versioning and improved dependency visibility. Resources Mentioned: Guy Dassa - https://www.linkedin.com/in/guydassa/ 15Five - https://www.linkedin.com/company/15five/ Apache Airflow - https://airflow.apache.org/ MLflow - https://mlflow.org/ DBT - https://www.getdbt.com/ Kubernetes - https://kubernetes.io/ RedShift - https://aws.amazon.com/redshift/ 15Five - https://www.15five.com/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

    20 min
See All (37)

5
out of 5
20 Ratings

About

Welcome to The Data Flowcast: Mastering Airflow for Data Engineering & AI — the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward. Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. Podcast Webpage: https://www.astronomer.io/podcast/

Information

You Might Also Like

To listen to explicit episodes, sign in.

Stay up to date with this show

Sign in or sign up to follow shows, save episodes, and get the latest updates.
Select a country or region

Africa, Middle East, and India

Asia Pacific

Europe

Latin America and the Caribbean

The United States and Canada