The Data Flowcast: Mastering Airflow for Data Engineering & AI
Data orchestration and machine learning are shaping how organizations handle massive datasets and drive customer-focused strategies. Tools like Apache Airflow are central to this transformation. In this episode, Vasyl Vasyuta, R&D Team Leader at Optimove, joins us to discuss how his team leverages Airflow to optimize data processing, orchestrate machine learning models and create personalized customer experiences.
Key Takeaways:
(01:59) Optimove tailors marketing notifications with personalized customer journeys.
(04:25) Airflow orchestrates Snowflake procedures for massive datasets.
(05:11) DAGs manage workflows with branching and replay plugins.
(05:41) The "Joystick" plugin enables seamless data replays.
(09:33) Airflow supports MLOps for customer data grouping.
(11:15) Machine learning predicts customer behavior for better campaigns.
(13:20) Thousands of DAGs run every five minutes for data processing.
(15:36) Custom versioning allows rollbacks and gradual rollouts.
(18:00) Airflow logs enhance operational observability.
(23:00) DAG versioning in Airflow 3.0 could boost efficiency.
Resources Mentioned:
https://www.linkedin.com/in/vasyl-vasyuta-3270b54a/
Optimove -
https://www.linkedin.com/company/optimove/
https://airflow.apache.org/
https://www.snowflake.com/
Datadog -
https://www.datadoghq.com/
https://astronomer.typeform.com/airflowsurvey24
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning