I enjoyed this write up a ton, thank you Mihail! As you look further into #Airflow, did you have a look at #Prefect or #Dagster? The more you use it for data, I truly believe these tools are a better fit for it. Especially dagster integrates with #Kubernetes out-of-the-box and spawning new jobs in seperate pods. In case you want to know more, I wrote about getting hands-on with dagster: https://sspaeti.medium.com/building-a-data-engineering-project-in-20-minutes-85c37cad4d87