Highly recommend to checkout dagster as it solves many higher-level problems that above orchestration tools most probably will not be able to solve as thought without the higher-level abstraction. E.g. testing locally and on production seems impossible as you mentioned, especially with spark, cloud DWHs, and many other hard-to-test components. But with dagster it's possible, and it's especially well tough through with resources, run-configs, mocks, and much more. Some more reasons and architecture overview I presented on the community meetings of dagster in case of interest: https://m.youtube.com/watch?v=HRd6rEU33XM&t=86s