Composable Data Systems
Created:
Wes McKinney - The Road to Composable Data Systems: Thoughts on the Last 15 Years and the Future; Pedreira, Pedro, Orri Erling, Konstantinos Karanasos, Scott Schneider, Wes McKinney, Satya R Valluri, Mohamed Zait, and Jacques Nadeau. “The composable data management system manifesto,” 2023. 16 (10)https://www.vldb.org/pvldb/vol16/p2679-pedreira.pdf. PDF
The stack proposed by this article:
- apache-arrow – language-independent compute and data interchange layer and supporting systems infrastructure across important programming languages
- ibis and dplyr – backend-agnostic data frame interfaces
- RAPIDS – GPU-accelerated libraries for data analytics and machine learning
- duckdb and velox – embeddable systems providing fast columnar query processing
- Substrait – - language-independent intermediate representation (IR) middleware for analytical computing to assist in decoupling user interfaces from compute engines