Data Engineering Created: Feb 20, 2018 by Pradeep Gowda Updated: Feb 07, 2024 Tagged: data-engineering 2022-10-24: The Evolution of The Data Engineer: A Look at The Past, Present & Future | Airbyte; via HN 2020-12-14: What Is Data Engineering and Is It Right for You? – Real Python 2024-02-07: A tale of three data platforms – The convergent evolution of BigQuery, Snowflake and Databricks demonstrates a broader tendency in the development of cloud platforms Software and Libraries Weld is a runtime for improving the performance of data-intensive applications. It optimizes across libraries and functions by expressing the core computations in libraries using a small common intermediate representation, similar to CUDA and OpenCL. Naiad is a distributed system for executing data parallel, cyclic dataflow programs. It offers the high throughput of batch processors, the low latency of stream processors, and the ability to perform iterative and incremental computations. frankmcsherry/timely-dataflow: A modular implementation of timely dataflow in Rust DataFusion: Big Data Platform for Rust Articles Take the tools out of ‘Data’, but don’t take the data out of the tools Books 📖 Data Engineering Design Patterns (DEDP); WIP; Dec 2023. The discussion at HN around how the definition of data engineer has changed is an interesting one.