Nicely written David, thanks for eluting the problem that many have or will have in the future. I went through the same. We solved the issue at our company with dagster.io and PyPi packages.

A perfect example how to use the advantages of both worlds, being ad-hoc experimenting and coding in notebooks and get production-ready and testable pipelines with dagster by the Earnest team: https://youtu.be/oCakb_tB_dU?t=448