Feb 22, 2022
Thanks Doug for sharing. I think everyone knows Python is slow, but speed isn't all it matters. I think in data engineering, many are using pandas with NumPy and other libraries that are written in C and Cython. What we already see is libraries written in Rust such as datafusion or integration for delta lake. I'm pretty sure we will see more like this. With airflow, dagster and and and, I still believe it will the language of choice for a long time. But the slow parts will be replaced with fast libraries.