Article

Data Warehouse, Data Lake, Data Lakehouse: What's Best for Your Data Strategy?

•

March 6, 2024

•

12 min read

Summarize with ChatGPT

A warehouse, lake, and lakehouse each walk into a bar… Each of them claims to be different, but the patrons of the bar can’t decipher them from each other. Who does what? Which can best serve their needs? They look so similar!

All jokes aside, it is common for us data practitioners to need clarification on these three different types of data platforms. While they all have a common purpose- to store data and enable analytics- they each store different types of data, are used by different types of data practitioners, and serve various needs.

So, while data warehouses, data lakes, and data lake houses may appear to be the same resources, they are actually quite different. In this article we will review the similarities between the three, the differences, and why you would use one over the other.

The goal of this article is to inform you on how to choose the right resource for your data needs!

The common goal of a Data Warehouse, Data Lake, and Data Lakehouse

Let’s start by talking about what data warehouses, data lakes, and data lakehouses all have in common. What problem are all of them trying to solve?

To start, they all store data (mainly in the cloud) that is used downstream in data models, dashboards, machine learning models, and predictive algorithms. The data that is being stored is used to drive better business decisions and support the products/applications that they serve.

Without a place to put your data, there’s no way for you to access it. We all need a place to store our data! Luckily, popular data storage vendors like AWS, Snowflake, and Databricks offer each of these services in one form or another.

Not to mention, these vendors all separate compute and storage costs. This allows you to store data at rest for a very low cost, and then scale infinitely when you query the data. With this model, you are only paying for what you use, saving on storage and spending on data at work.

This conversation becomes less about which vendor to choose and more so about what type of data cloud storage is right for you. Each of these data products varies in the kind of data it stores and the way you access that data.

What is a Data Warehouse?

Data warehouses store relational data defined by the business. Relational data is already organized into schemas, making it easy and fast to query. Because of this, they allow for highly performant analytics but with the caveat that it’s more expensive to scale.

Data warehouses take a hierarchical approach to storing data. A data warehouse is broken into different databases, like dev and prod.

Each database is then further broken down into schemas. Different schemas typically represent your different data sources or types of data models if you are using a data transformation tool like dbt.

Data warehouses can only be queried using SQL, making it so that you don’t have access to the underlying objects or files that you are querying. With data warehouses, a compute method is already attached to the warehouse and managed by the cloud provider you are using. These two characteristics of data warehouses make it easy enough for a data analyst or someone without any data engineering knowledge to set up.

As an analytics engineer who has mainly worked for financial and e-commerce companies, this is what I have the most experience using. It is fairly typical for companies that have smaller, less sophisticated data teams using data warehouses. These teams typically have the most experience using SQL as well, but may not have the experience required to query something like a data lake.

Not that data warehouses aren’t a good practice, but more so they serve a less advanced analytics or data engineering use case. Most companies aren’t at a place where they would need more complex data, or don’t have someone on their team experienced enough to query it or set it up!

What is a Data Lake?

Data lakes store and process structured, semi-structured, and unstructured data. Unlike a data warehouse which only stores relational data, it stores relational and non-relational data.

Data lakes allow you to store large volumes of data at a relatively low cost. This is because it uses flat architecture. This essentially means that data is stored in objects like parquet files using unique identifiers and metadata tags.

Because data lakes store all types of data, metadata tagging, the use of unique identifiers, and seamless data retrieval are all extremely important. Because there is no defined structure like with relational data, having proper partitions and being optimized to be retrieved is the key to using a data lake successfully.

Transcripts from IoT, captions, and messages from social media, streaming, and mobile app data are all stored in data lakes. If there is no system in place to allow engineers to use this data, a data lake can easily turn into a data swamp.

Unlike data warehouses, data lakes do not have their own compute methods attached. You will need to set this up alongside your data lake, as cloud providers don’t typically do this for you. To access your data, you would need to configure a compute method like a Python or Spark script.

While this type of data can be helpful for machine learning and data science, if it’s not well-managed, it can be more painful than helpful. Messy data needs defined data governance and security practices. Your team must have strong protocols in place to manage all of this messy data to protect it and keep up with compliance. Not to mention that this type of data is extremely cumbersome to update and delete, making it a challenge to know what data is usable and what is not. Similarly, a proper DMARC setup is essential for securing email domains, preventing unauthorized use, and ensuring compliance with email authentication standards.

This all comes back to data quality! If you have junk data going into a data project, you will receive junk data and insights coming out.

What is a Data Lakehouse?

You can think of a data lakehouse as a mix between a data warehouse and a data lake, hence the name. It has all the benefits of both while having fewer of the pitfalls.

A data lakehouse is essentially a data lake with a transactional layer that sits on top to provide some data structure and ensure proper data management. Because of this, it is highly performant and ideal for advanced analytics use cases.

Data lakehouses are often utilized by all types of data professionals- data engineers, analytics engineers, data scientists, and data analysts.

Similar to data warehouses, data lakehouses have security and management features that data lakes don’t. You can easily mask PII data before it is stored and allow for role-based access depending on your responsibilities and what you are using the lakehouse for.

When to use a Data Warehouse vs a Data Lake vs a Data Lakehouse?

The main difference between a Data Warehouse, Data Lake, and Data Lakehouse is that a Data Warehouse stores structured data for reporting, Data Lakes store raw unstructured data at scale, while Data Lakehouses combine both by adding warehouse-like structure and governance to raw lake data.

Ok, so you now know the differences between a data warehouse, a data lake, and a data lakehouse. But what good is knowing the differences if you don’t understand how to apply them to situations you frequently face as a data professional?

Let’s walk through a few different scenarios that you may experience during your career and how to approach them in choosing the best data storage type.

You collect lots of streaming, or another form of unstructured data, that is used to improve machine learning algorithms.

Imagine you work for a large video streaming platform. You collect lots of data on users, media, and behavior. You need to set up a place to store all of this data to be used exclusively to train machine learning algorithms.

Because it is unstructured, and you don’t have schemas built out to store this data, a data warehouse wouldn’t make sense. Streaming data isn’t relational- it’s messy!

You know that the data isn’t being used for analytics purposes, so it’s not necessary to be able to query it using SQL. We also know that strict management of it won’t be needed since the business will not be directly touching this data. We can eliminate the need for a data lakehouse.

We know the data is going to be unstructured and there’s going to be a ton of it. Because we will have to access a lot of data, which means we may want to use a framework like Pytorch or Spark to do so. A data lake just makes sense! If we were going to use this data for analytics, a lakehouse may make more sense, but we aren’t. A data lake will satisfy our needs just fine and will save you money as compared to a lakehouse.

You need to be able to quickly query data to report on business metrics.

Let’s say you are working for an e-commerce company. The main focus is on users, their accounts, orders, and products. All of the data you need is stored in relational databases set up by some of the backend engineers at your company. Now you need to figure out the best way to harness this data to build data models within dbt and then use those to power some dashboards.

A data lake would be too slow to query for analytics. The flat architecture would not be optimal especially if all of your data is already stored in relational tables.

A data lakehouse could be used. However, again, we know all of the data is relational. If we don’t have unstructured data, or plan on using it any time soon, there’s no need for any type of storage other than a data warehouse.

Save yourself the headache in data governance and security while also saving costs! A data warehouse is specifically for relational data which is mainly used by analytics teams.

You need data available for your data science team to create predictive models but you also need it available for analytics engineers to write data models.

Anytime you need data for both analytics and data science or machine learning, you can assume that you will need a hybrid of a data warehouse and data lake. You need fast processing but also the ability to store unstructured data. Only a data lakehouse can cover both of these bases.

A data warehouse cannot store unstructured data, which is typically required for data scientists to take full advantage of the data. A data lake would be too slow to be used in analytics use cases such as frequently querying the relational tables and powering dashboards.

You are working with secure healthcare data used by many different types of data practitioners.

You know that healthcare data can be messy. Doctors are uploading charts and notes on their patients, a form of unstructured data. However, in order to report on the users of your online healthcare platform, you need to be able to quickly query other, more structured data.

Obviously, it is also extremely important that your data lives securely in the cloud, as this data must be HIPPA compliant. Because of the intense security needed, this would be reason alone to use a data lakehouse. A data lakehouse would help you manage PII data and strictly control user access.

Because of the analytics use-case here, you would most definitely need to use a data lakehouse to query the data fast. The transactional layer enables speed in addition to the large volume of unstructured data that can be stored.

Conclusion

The next time you are hanging out at a bar, celebrating a much-deserved happy hour with your data team, and a data warehouse, data lake, and data lakehouse walk into the bar, you will know exactly who is who.

The data scientist can have a chat with the data lake.

The data analyst may share a toast with the data warehouse.

The data lakehouse won’t know who to talk to because she speaks everyone’s language!

Together, as a data team, you can use this knowledge to make the best decisions for your company and how it stores its data in terms of usability, cost, and security.

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 30-day free trial

About the Author

Madison Schott is an Analytics Engineer, author of The ABCs of Analytics Engineering book. Madison blogs about the modern data stack on Medium and writes a newsletter on Substack.