I find myself, again and again, asking if Rust will be for DE. The thing that delta lake and datafusion exist or invest in Rust is a good sign. But after all, Rust is a systems programming language and, therefore, compared to C++. I assume things that have to be fast, like delta lake, that interact with the file system and process lots of data, will be written in Rust. But as a data engineer, a level higher up speed is important, not that essential in most cases.

And on the other hand, it seems that Scala has more ideal tools that suit real fast data wrangling. But as you said, at the end of the day, you need to choose the tools that fit best, I agree—looking forward to how this will turn out.

--

--

--

--

Excellent write-up Doug, appreciate it! I also see it more and more in my promoted ads. I think they're now trying to get users. I haven't tried either; however, I don't like the proprietary F3 format, or do you know if you could also use Parquet, Delta or other formats?

I think you cannot compare it with Dremio as Demio is just a Data Visualisation tool with a lot of in-memory Apache Arrow, so not persisted. I categorised these tools in OLAP-Technologies (Druid, ClickHouse, ...), Cloud Data Warehouses (FireBold, Snowflake, BigQuery, ...), Data Visualisations (Dremio, Informatica Data Virtualisation, ...) and Serviced Cloud & Analytics (Looker, Sisense, ...). In case of interest, here I wrote about the different technologies: https://www.sspaeti.com/blog/olap-whats-coming-next/#list-of-cube-replacements

--

--

Simon Späti

Data Engineer & Technical Author with 15+ years of experience. I enjoy maintaining awareness of new innovative and emerging open-source technologies.