Xinran Waibel have you checked out dagster.io and dagstermill? I find this the best integration into pipelines. Check out: https://dagster.readthedocs.io/en/0.3.3.post0/intro_tutorial/dagstermill.html