The Data Engineer: A Dead Man Walking? Charles Giardina recently gave an insightful presentation on the future of data engineering, boldly claiming that the data engineer is dead and the industry just hasn't realized it yet. He highlighted the emergence of a clean interface between data movers (data engineers) and data users (data analysts, data scientists, and analytics engineers). This interface, known as the data movement interface, allows data users to specify what data they need and where it needs to go, while the data movers make it happen.
The Historical Context: ETL to ELT Giardina took us on a journey through the history of data engineering, starting with the Hadoop era, where ETL (Extract, Transform, Load) was the standard approach for data movement. However, with the rise of data warehouses, the paradigm shifted towards ELT (Extract, Load, Transform), enabling data users to transform and analyze data themselves without relying on software engineers.
The New Abstraction: ELT and the Future of Data Engineering In the era of ELT, data users can handle data movement themselves, reducing the need for data engineers. Giardina believes that data engineers are currently fighting against decades of momentum, but the existing interface between data users and data movers can eventually replace them.
The Unresolved Issues: Security and Incident Management However, Giardina also acknowledges two areas where the ELT paradigm has yet to fully address concerns: information security and compliance, and incident management. As data users are not best positioned to tackle these challenges, there is still room for improvement in ELT tools to integrate security and compliance features, as well as to handle incidents effectively.
The Bright Side: A Positive Outcome for Data Engineers Despite his bold claims, Giardina's vision is not a bleak one for data engineers. As the ELT paradigm makes data movement easier, the pace of innovation in the data industry will accelerate, and data engineers will be able to shift into other parts of the modern data stack. In short, the death of ETL could lead to a positive-sum outcome for data engineers and the data ecosystem as a whole.
In conclusion, Charles Giardina has sparked a thought-provoking discussion on the future of data engineering and the potential obsolescence of ETL. As the industry continues to evolve, data engineers must adapt and explore new roles within the modern data stack.
