Simon Späti
Feb 22, 2022

--

Thanks Doug for sharing. I think everyone knows Python is slow, but speed isn't all it matters. I think in data engineering, many are using pandas with NumPy and other libraries that are written in C and Cython. What we already see is libraries written in Rust such as datafusion or integration for delta lake. I'm pretty sure we will see more like this. With airflow, dagster and and and, I still believe it will the language of choice for a long time. But the slow parts will be replaced with fast libraries.

--

--

Simon Späti
Simon Späti

Written by Simon Späti

Data Engineer & Technical Author with 15+ years of experience. I enjoy maintaining awareness of new innovative and emerging open-source technologies.

Responses (1)