I find myself, again and again, asking if Rust will be for DE. The thing that delta lake and datafusion exist or invest in Rust is a good sign. But after all, Rust is a systems programming language and, therefore, compared to C++. I assume things that have to be fast, like delta lake, that interact with the file system and process lots of data, will be written in Rust. But as a data engineer, a level higher up speed is important, not that essential in most cases.

And on the other hand, it seems that Scala has more ideal tools that suit real fast data wrangling. But as you said, at the end of the day, you need to choose the tools that fit best, I agree—looking forward to how this will turn out.

--

Data Engineer & Technical Author with 15+ years of experience. I enjoy maintaining awareness of new innovative and emerging open-source technologies.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Simon Späti

Data Engineer & Technical Author with 15+ years of experience. I enjoy maintaining awareness of new innovative and emerging open-source technologies.