Wes McKinney - The Road to Composable Data Systems: Thoughts on the Last 15 Years and the Future; (Pedreira et al., 2023) PDF
The stack proposed by this article:
- apache-arrow — language-independent compute and data interchange layer and supporting systems infrastructure across important programming languages
- ibis and dplyr — backend-agnostic data frame interfaces
- RAPIDS — GPU-accelerated libraries for data analytics and machine learning
- duckdb and velox — embeddable systems providing fast columnar query processing
- Substrait — - language-independent intermediate representation (IR) middleware for analytical computing to assist in decoupling user interfaces from compute engines
Pedreira, P., Erling, O., Karanasos, K., Schneider, S., McKinney, W., Valluri, S. R., Zait, M., & Nadeau, J. (2023). The Composable Data Management System Manifesto. Proceedings of the VLDB Endowment, 16(10), 2679–2685. https://www.vldb.org/pvldb/vol16/p2679-pedreira.pdf