Nice, I do understand all of this, I made my way through the same transition several times as well (SSIS and so on ;-)

Curious, did you write all the code yourself? If so, did you consider using open-source orchestrators as Airflow, Prefect or Dagster? If yes, why you didn't use it? If not, then I suggest having a look. I have some articles about it as well in case of interest!