Data Engineering

News

• 2022-10-24: The Evolution of The Data Engineer: A Look at The Past, Present & Future | Airbyte; via HN

• 2020-12-14: What Is Data Engineering and Is It Right for You? – Real Python

• 2024-02-07: A tale of three data platforms – The convergent evolution of BigQuery, Snowflake and Databricks demonstrates a broader tendency in the development of cloud platforms

Software and Libraries

• Weld is a runtime for improving the performance of data-intensive applications. It optimizes across libraries and functions by expressing the core computations in libraries using a small common intermediate representation, similar to CUDA and OpenCL.
• Naiad is a distributed system for executing data parallel, cyclic dataflow programs. It offers the high throughput of batch processors, the low latency of stream processors, and the ability to perform iterative and incremental computations.
• frankmcsherry/timely-dataflow: A modular implementation of timely dataflow in Rust
• DataFusion: Big Data Platform for Rust

Articles

• Take the tools out of ‘Data’, but don’t take the data out of the tools

Books

• 📖 Data Engineering Design Patterns (DEDP); WIP; Dec 2023. The discussion at HN around how the definition of data engineer has changed is an interesting one.
• Data Engineering With Rust
• Data Engineering Cookbook

Courses

Data Engineering Concentration - Data Science, MS | University of San Francisco