You just graduated college, or maybe you are looking for a career change after working in a certain role for a while. You know you are interested in data and solving problems but aren’t sure which career would be right for you. Should you become a data analyst? Or maybe a data engineer?
Data analysts lean more toward the business side of things, creating reports and dashboards to present to stakeholders. Data engineers lean more toward coding in Python and deploying important infrastructure to support data collection. But what if I told you that you could have both? This is where the role of the analytics engineer comes in.
What is analytics engineering?
Analytics engineers sit in between data analysts and data engineers. They mainly code in SQL but also code in Python and work with Business Intelligence (BI) tools, the best of both worlds. Analytics engineers own the data pipeline from ingestion to transformation, as well as through the orchestration of this entire pipeline. They set up different tools to ensure this collected data is being moved from the backend, or from external sources, to the data warehouse.
Analytics engineers typically use a tool called dbt (or data build tool) to transform raw data into data models that can be automated and deployed within the same warehouse. These data models also include documentation and testing to ensure data quality is high. You could think of the analytics engineer as the data quality policeman.
Once data is transformed into data models that can be used by the business, the data analyst will then take over and ingest these models into their BI tool of choice. It is then their job to visualize the data how stakeholders want it. I like to think of the data analyst as the analytics engineer’s stakeholder!
5 signs you’d enjoy analytics engineering
Still debating whether analytics engineering could be the right career for you? Here are five signs it could be the best move for you to make.
You enjoy solving strategic problems by looking at data (and writing SQL queries).
Analytics engineering is all about exploring the data you have available to you. And, in order to explore that data, you’ll be writing a lot of SQL queries. You need to understand how the data relates to each other and discover the unique edge cases so you can model the data in a way that makes sense. After all, the end goal is to produce a dataset that powers dashboards and reports that the data analysts are building.
Often times you’ll be the first to discover problems in the data or potentials that the business hasn’t even discovered. You need to be proactive about documenting these and fully understanding the problem at hand so you can present it to business teams. This often involves a lot of validation queries and bouncing possible ideas off the data analyst.
You have interest or experience in the business side of things.
Unlike data engineers, analytics engineers interact closely with stakeholders on product, growth, and marketing teams. Instead of interacting through a middleman like a product manager, you are speaking directly with those using the data every day to make informed business decisions. Because of this, you need to understand how these business teams function. You need to have experience with KPIs and generating them from the data you are dealt.
This was the factor that really drove me to want to be an analytics engineer. I was tired of being told what to do and having no say in how I thought it would affect the business. Being an entrepreneur and marketing major, I wanted that close interaction! I like knowing the why behind what I’m building.
Defining KPIs within my data models gives me that business exposure that I craved when I was a data engineer. As an analytics engineer, you build modular data models to define these on the right granularity, optimizing the way data is delivered to business teams. You talk with the teams to understand when they need the data, how they want to filter it, and why it’s important to them- all contexts you don’t receive as a data engineer.
You are a leader who paves the way by setting best practices and standards for your team.
If you like things done your way, analytics engineering is a great career for you. It's a career where you can take charge of standards being set in your data. Analytics engineers create a style guide to govern how models are built and how the data pipeline is set up. This style guide typically contains standards like how you should be naming your models and columns, best practices for making code changes, and how often your data pipeline should run.
Documentation is your best friend. Analytics engineers are the ones on top of data documentation within the engineering and data teams. They ensure data sources, data models, and dashboards are all properly documented so that the data environment is easy for everyone to navigate. Data tends to go undocumented and be quite messy without an analytics engineer, which you will probably see if you join as the first one on a team.
You’re a stickler about good-quality data.
Chances are if you care about standardization and documentation, you also care about data quality. Analytics engineers are the ones to test and validate data models, ensuring they meet high standards. They look into different factors like freshness, completeness, and accuracy to ensure downstream data models are not breaking.
This is one of the most important duties of a good analytics engineer. Because, at the end of the day, if you have bad-quality data that is being used to make business decisions, those decisions aren’t going to have a great outcome. Bad data in equals bad data out. It’s important that you add tests in your data models to catch issues as soon as possible.
As part of data quality, it’s imperative that you properly validate all of your data models so that you can be confident in the KPIs you are producing. When you properly validate, business teams don’t have the option to second-guess the numbers they are seeing. This builds trust in the data, something that is surprisingly hard to have.
You enjoy discovering new tools and ways of doing things.
One of the coolest parts of being an analytics engineer is all the new tools you get to discover. The modern data stack is still a fairly new concept and new tools are popping up every day. The market is constantly evolving and I am always looking for new technologies to use for warehousing, ingestion, orchestration, testing, and observability. There is always something new disrupting the space!
Some of my favorite tools to discover are open-source ones. These are great because they are free, saving money for your company while also improving its data processes. Airbyte is an open-source data ingestion tool that I first discovered when looking for an affordable way to move data from external sources to my Snowflake data platform. I also love dbt for data transformation and the free packages that you can download with it such as re_data and dbt expectations. Another favorite is Prefect for orchestrating data pipelines using simple Python code.
Finding new tools is like a treasure hunt. There is always something new to discover that can bring immense value to your company whether that’s through saving money, producing higher-quality data, or saving the team valuable time. So, if you like learning new things and constantly improving what you are building, analytics engineering could be for you!
Conclusion
Data engineers work on back-end infrastructure that helps collect data to be used by analytics engineers. Analytics engineers interact with business teams to define KPIs and produce data models with the right columns and metrics. Data analysts then use the datasets produced by these data models to create reports and dashboards for stakeholders to use.
Analytics engineers own the following tasks:
- Setting up the modern data stack from ingestion to orchestration using the appropriate tools
- Data transformation within data models
- Building data pipelines and orchestrating data models
- Setting and documenting standards across the data
- Maintaining high-quality data
- Working with business teams to properly define KPIs
If these tasks interest you, or if you’ve done them in the past, you may be the perfect fit for analytics engineering! If you’re interested in learning more about the career, be sure to check out my new ebook “The ABCs of Analytics Engineering”, a learning resource that walks you through important concepts to know as an analytics engineer. This book will teach you everything from how to craft the perfect resume to what tests to implement for high-quality data.
About the Author
About the Author
%201.svg){kind=small}