Simon Späti
Mar 15, 2021

--

Nice write up and well explained! But I don't agree with your conclusion, I would guess you end up in a chaos if you use trigger, trigger, trigger.

I agree that Jupyter notebooks are easy for people not engineers. But with Papermill and Dagster, you can easily integrate them into your orchestrator or pipelines: https://towardsdatascience.com/business-intelligence-meets-data-engineering-with-emerging-technologies-8810c3eed8b1

I would also have a look to alternatives to Airflow as Airflow is not data driven, but execution driven: https://www.quora.com/What-are-common-alternatives-to-Apache-Airflow

Plus triggers are also supported by orchestrators and set-up with Kubernetes is also getting easier and easier.

--

--

Simon Späti
Simon Späti

Written by Simon Späti

Data Engineer & Technical Author with 15+ years of experience. I enjoy maintaining awareness of new innovative and emerging open-source technologies.