Simon Späti

I find myself, again and again, asking if Rust will be for DE. The thing that delta lake and datafusion exist or invest in Rust is a good sign. But after all, Rust is a systems programming language and, therefore, compared to C++. I assume things that have to be fast, like delta lake, that interact with the file system and process lots of data, will be written in Rust. But as a data engineer, a level higher up speed is important, not that essential in most cases.

And on the other hand, it seems that Scala has more ideal tools that suit real fast data wrangling. But as you said, at the end of the day, you need to choose the tools that fit best, I agree—looking forward to how this will turn out.

--

--

This article is excellent, and I like that you compare Elixir with Scala and Clojure. Especially as a Data Engineer, I'm constantly thinking about my next language to master besides Python. I fancied Clojure, but it's somewhat still hard to use for many use cases. I see more functional programming…

--

--

Excellent write-up Doug, appreciate it! I also see it more and more in my promoted ads. I think they're now trying to get users. I haven't tried either; however, I don't like the proprietary F3 format, or do you know if you could also use Parquet, Delta or other formats?

--

--

Simon Späti

Simon Späti

Data Engineer & Technical Author with 15+ years of experience. I enjoy maintaining awareness of new innovative and emerging open-source technologies.