Thanks, Kris for this article. I share the enthusiasm about #dbt and the reasoning behind. But I wouldn't compare it to Spark; it's a different thing IMO. In my eyes, dbt is an #orchestrator that talks #SQL. Therefore it runs Spark jobs, even more with #deltalake having a native SQL-API that runs Spark in the background.

That's why I would compare it to Apache #Airflow, #Dagster, #Prefect, etc. (https://qr.ae/pNrIPi). And in that perspective, I like to see dbt more often, especially in combination with dagster as a higher level orchestrator for none SQL matters. Dagster has a first-class integration of dbt: https://dagster.io/blog/dagster-dbt. And you can see a real-life crazy good example of what is possible if you have a heterogeneous landscape (which you always have) with, e.g. Stich, MongoDB, SQLServer, S3, GoogleSheets, Snowflake, Jupyter notebooks, dbt: https://dagster.io/blog/good-eggs-3 (credits to Goodeggs and dwall!)