The Data Ecosystem Is Ready for ETL To Be Dead

Data

March 24, 2023

5 min read

Charles Giardina

Summarize with AI:

‍

The Data Engineer: A Dead Man Walking?

Charles Giardina recently gave an insightful presentation on the future of data engineering, boldly claiming that the data engineer is dead and the industry just hasn't realized it yet. He highlighted the emergence of a clean interface between data movers (data engineers) and data users (data analysts, data scientists, and analytics engineers). This interface, known as the data movement interface, allows data users to specify what data they need and where it needs to go, while the data movers make it happen.

‍

The Historical Context: ETL to ELT

Giardina took us on a journey through the history of data engineering, starting with the Hadoop era, where ETL (Extract, Transform, Load) was the standard approach for data movement. However, with the rise of data warehouses, the paradigm shifted towards ELT (Extract, Load, Transform), enabling data users to transform and analyze data themselves without relying on software engineers.

‍

The New Abstraction: ELT and the Future of Data Engineering

In the era of ELT, data users can handle data movement themselves, reducing the need for data engineers. Giardina believes that data engineers are currently fighting against decades of momentum, but the existing interface between data users and data movers can eventually replace them.

‍

The Unresolved Issues: Security and Incident Management

However, Giardina also acknowledges two areas where the ELT paradigm has yet to fully address concerns: information security and compliance, and incident management. As data users are not best positioned to tackle these challenges, there is still room for improvement in ELT tools to integrate security and compliance features, as well as to handle incidents effectively.

‍

The Bright Side: A Positive Outcome for Data Engineers

Despite his bold claims, Giardina's vision is not a bleak one for data engineers. As the ELT paradigm makes data movement easier, the pace of innovation in the data industry will accelerate, and data engineers will be able to shift into other parts of the modern data stack. In short, the death of ETL could lead to a positive-sum outcome for data engineers and the data ecosystem as a whole.

In conclusion, Charles Giardina has sparked a thought-provoking discussion on the future of data engineering and the potential obsolescence of ETL. As the industry continues to evolve, data engineers must adapt and explore new roles within the modern data stack.

‍

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->