Data Engineering to Analytics Engineering: How to Successfully Transition

Transitioning from Data Engineering to Analytics Engineering made easy. Navigate your career shift with confidence!

DATA

May 9, 2023

10 min read

Madison Schott

Summarize with AI:

Data Engineering to Analytics Engineering: How to Successfully Transition

A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded. I was beginning to realize that I wasn’t as passionate about my work as I had hoped.

I think it’s important to follow your interests, no matter where you are, in order to find out what you actually want to do. This can mean pursuing hobbies outside of your 9-5 that bring you joy, or getting involved with different teams at the company you already work at.

I remember how much I enjoyed marketing and the business side of things in college, so I decided to start exploring my options there. I began talking with data analysts who solved business problems every day using data. They were like data engineers with way more business exposure!

By asking for data analytics projects, I learned how to use dbt and further developed my Python skills. Playing around with new technologies helped me to see that I still enjoyed working in data, I just had to be using the right tools and solving the right problems. Eventually, this led to me seeking out a different role that was more aligned with these new-found interests and skill sets. This role happened to be that of an analytics engineer.

How to know you should transition to analytics engineering

A lot of people are scared to make the transition from data engineer to analytics engineer because they don’t know if the time is right. The truth is, there will never be a “right” time to do anything. However, if you’re feeling unchallenged and unfulfilled, you can never make this choice too soon.

Transitioning from data engineering to analytics engineering is probably right for you if you are curious about the data itself rather than building products that support the data. Analytics engineering is way more customer-oriented because you are using your customers’ data to answer key business questions. You are more so focusing on growing revenue and insight rather than building things.

This brings me to my second point. If you’re wanting more of a connection to the business, and want to be making decisions that drive growth for the company, analytics engineering may be the move for you. As a data engineer, you are assigned tasks by a project manager. You don’t necessarily have a say in what you’re solving or what you think needs to be prioritized. But with analytics engineering, you do.

How I discovered analytics engineering

Truth be told, I had no idea what analytics engineering was when I realized I wanted to switch data roles. I thought my only option for working closer to the business was to become a data analyst. And that’s exactly what I tried to do.

I applied for so many data analyst roles, but wasn’t having much luck. I lacked a lot of that deep business experience needed by an analyst, as well as the skills to build out proper dashboards. Rather than on job title, I started to focus on roles that had a mix of skills that I already had and those that I wanted to learn.

Eventually, I stumbled upon the role of an analytics engineer. This role required knowledge of SQL, Python, AWS, orchestration, dbt, and data warehousing, all skillsets I had gained as a data engineer. However, it also required experience working with modern data stack tools, communicating with business teams, and some basic BI reporting.

While I hadn’t had much modern data stack experience, I was lucky to find a company that believed in my passion and desire to learn. Sometimes if the fit is right, the company is willing to hire you without all of the requirements!

Skills to develop before transitioning to analytics engineering

There are a few skills I spent time really developing before seeking out an analytics engineering role. These are the skills that are most essential and will give you a leg up over other candidates. If you focus on these three skills, I’m confident that you can easily teach yourself on the job the other skills required to be a good analytics engineer.

dbt

dbt, or data build tool, is the data transformation tool that really carved the way for analytics engineers. In fact, the company behind it is the one who coined the name analytics engineer! While you don’t necessarily have to know dbt to be an analytics engineer, it's a skill a lot of companies are looking for when hiring for the role.

dbt is a SQL-based tool, so if you already know SQL, it is fairly easy to learn. You would need to familiarize yourself with setting up a dbt project, data modeling best practices, and the purpose of each type of dbt data model. I recommend checking out dbt style guides for an outline of the “do”s and “don’t”s of writing SQL code in dbt. This will help you learn the standards that you may already see in place when you join a team.

dbt also uses a templating language called Jinja for documentation and functions within the tool. However, instead of calling them functions, dbt calls them macros. Macros are a more advanced feature and are the equivalent to functions. You can use them to automate SQL output within your data models. While this isn’t necessary to land a job, it’s definitely a helpful skill to make your life of an analytics engineer easier.

Business Communication

This is probably the most difficult skill to develop and practice as a data engineer. We are so used to talking technically with other engineers that we forget to explain things in an easy-to-understand way. When communicating with business teams, you need to know how to explain technical concepts to a non-technical audience. The simpler the better. The crowd that you are talking to is almost never going to be as technical as you are.

In addition to this, you need to understand the business lingo that these teams speak. You will constantly hear different metrics like CAC, MRR, NPM, and ROI. Make sure you know what these mean! Chances are, these are the reasons you are building a data model in the first place. You need to understand the end goals/metrics and the data that goes into calculating them.

Building Data Pipelines

Lucky for you, as a data engineer, you most likely already have this skill! Building data pipelines is a common skill of many data engineers that can also be used as an analytics engineer. However, in this context, it’s important you know how to orchestrate different dbt models with various data sources. You need to be able to deal with dependencies from a bunch of different source systems.

Common data pipeline tools in analytics engineering include Airflow, Prefect, and Dagster. These are all built using simple Python, another important skill to have for building out a pipeline. Pick one tool and familiarize yourself with it! Once you learn one, you most likely can learn them all.

What I wish I knew before transitioning into analytics engineering

Of course, looking back and transitioning from data engineering to analytics engineering, there are some things I wish I knew that would have made my journey a lot more enjoyable.

Your data engineering skills will still be used every day and extremely valuable.

When transitioning from a data engineer to an analytics engineer, it’s easy to think that your entire role and skill set will be changing. However, there is a ton of overlap between analytics engineering and data engineering! Oftentimes, the skills you learn as a data engineer become your biggest superpower. You can probably do things other analytics engineers can’t, so embrace it!

I actually think it gives you a leg up over other candidates when applying for roles, especially if you are looking for a data role at a small company who is just beginning to hire out its data team. Chances are, they are looking to get the most bang for their buck, and want someone who can do a little bit of everything. An eager data engineer turned analytics engineer is just the person for the job!

You don’t have to know everything in the beginning.

It can be easy to get caught up in needing to learn everything all at once. I need to know how to set up an open-source data connector like Airbyte before I apply for that role. I need to have experience building custom macros in dbt before I appy for that role. I can’t apply for that role until I master every type of SQL window function.

Instead of focusing on what you don’t know, focus on what you do! What makes you unique? Why would someone want to hire you over someone else? Think about the things that can’t be learned on the job that makes you stand out. The industry is always changing, meaning there’s always going to be something you don’t know. Embrace it! That’s one of the best parts about analytics engineering.

When I started as an analytics engineer, I had no idea what dimensional modeling was. I was actually introduced to it through dbt’s documentation when I was first experimenting with organizing our data warehouse and modeling our data. Eventually, I was able to learn what that was, apply the techniques to my work, and improve my data modeling skills. Now I can talk to you for days about dimensional modeling!

It’s never too early to start applying for new roles.

If you’re unhappy, do something about it. The longer you sit around in a career that is not serving you, the more you miss out on opportunities to learn and grow. Every time I’ve made a career switch it was because I was no longer being challenged by my work. If we aren’t learning new things and growing as a person, we are staying the same. And is there anything worse than that?

Even if you think you don’t have the right skills right away, you will learn by exploring your options. Browse different roles and see what companies are looking for. What are the commonalities in skills between job postings? Is there anything you think you wouldn’t like about an analytics engineering role? Looking at job descriptions is a great way to find this out!

Conclusion

Now looking back, I couldn’t have made a more amazing transition from data engineering to analytics engineering. Deep down I knew that a purely technical role just wasn’t right for me. I needed to work with data hands on, using it to solve customer and business problems.

Life is too short to stay in a role you aren’t fully enjoying! Don’t be afraid to explore what else is out there whether that is analytics engineering or not. What’s the worst thing that can happen? If you make a jump and realize that actually isn’t for you, you can always go back to your old role. But, you will never know unless you try!

Suggested Read

Data Engineering Roadmap

‍

About the Author

Madison Schott

Madison Schott is an Analytics Engineer, author of The ABCs of Analytics Engineering book. Madison blogs about the modern data stack on Medium and writes a newsletter on Substack.