The role of an analytics engineer is a term that has largely emerged due to the extensive use of dbt, where software engineering merges with business expertise. With the recent emphasis on the semantic layer and the acquisition of Transform, it's intriguing to see the direction in which dbt is taking this concept to the next level. Below, you'll find highlights, comments, and thoughts on The Next Big Step Forward for Analytics Engineering
Tristan Handy from dbt reflects on the progress made since the manifesto's introduction in 2016
Tristan guides us through the history of software engineering, illustrating how the industry has evolved from individual "heroic" efforts to today's efficient, team-based approach. This evolution led to the adoption of service-oriented architecture (SOA) and "two-pizza teams," which consist of 5-9 people who own their codebase and manage its lifecycle.
The article then compares the current state of the dbt ecosystem with the software engineering world, noting that dbt projects have become more complex, but the tools and processes are still lagging behind. Tristan identifies three key capabilities missing from dbt: the ability to define interfaces, define and validate contracts
To address these gaps, Tristan reveals that dbt Core v1.5 will introduce new constructs, including Access , Contracts , and Versions dbt semantic layer acquisition
In the future , Tristan envisions individual teams owning their data, with each team managing their codebase, warehouse spending, and analytics tools. This will ultimately lead to improved scalability and efficiency in the analytics world. Tristan expresses excitement for this future and encourages readers to share in his enthusiasm.
