6 ways to reduce Snowflake costs

Understand Snowflake pricing model to optimize Snowflake compute, storage and transfer costs.

DATA

July 26, 2022

10 min read

Madison Schott

Summarize with AI:

If you work for a smaller company, or even a start-up, then you know every dollar matters. Even in larger companies, departments are always looking for ways to reduce spend so that they can put their money into larger growth initiatives. Every few dollars saved in one tool can be put towards a new tool that will uplevel the modern data stack or even another brain to deepen analysis. Dollars are valuable!

Costs can be saved in many different ways. You can switch from an enterprise data ingestion tool to an open-source tool. You can choose to downsize your data team from 5 people to 3 people. Or, you could optimize the resources you are already using in order to save costs while keeping your current data stack and team the same. Personally, I’d choose this option.

If you are a Snowflake user like myself, then you will be happy to know that there are many different ways you can save Snowflake costs without compromising the flow of your data. We will get into how to do exactly this but first let’s review some key components of how Snowflake pricing functions. Understanding Snowflake pricing model will allow you to optimize resources and compute power.

Snowflake pricing: How Snowflake costs are calculated

Snowflake pricing is based on your compute, storage, and cloud services usage. Compute costs are calculated depending on how long your warehouses are running. Storage costs are calculated based on the amount of data you are storing. The cloud services depend on the Snowflake features you are using. In this section, we will dive deeper into how Snowflake charges you for each of these components.

Compute costs

A virtual warehouse is simply a cluster of compute resources. Each warehouse provides the necessary resources such as CPU, memory, and temporary storage, to complete a task. Warehouses come in sizes x-small to 6X-large, doubling in cost and compute power between each tier. Your Snowflake compute costs depend on the amount of time that the warehouse is continually running, or “online”.

When the warehouse is running, it is available to compute resources. This means that if your warehouse runs for 10 minutes at the top of each hour, you will only be charged for those 10 minutes rather than the whole hour. A warehouse is spun up automatically when you run a query. However, it is not shut down automatically unless you set an auto-suspend period. Later in the article, we will discuss how to ensure your warehouses aren’t continually running and shut off.

Storage costs

Many people are surprised to know that Snowflake’s storage costs are actually pretty low. Snowflake storage costs are based on the daily average amount of data stored across database tables, clones, and failover regions. This cost is relatively low compared to that of virtual warehouses. When optimizing costs, it’s important to focus on the latter rather than storage. Plus, you probably don’t want to delete any historic data that could be of value to the business.

Data transfer costs

There can be a few other Snowflake costs depending on the features you and your team are utilizing like data transfer costs. If you are transferring data from one Snowflake region to another, or to another cloud data platform, that will cost a per-byte fee.

Serverless features costs

Snowflake also charges for serverless features such as Snowpipe, a file-loading service. However, you are charged for actual compute usage since Snowflake manages the warehouses here, rather than the customer-managed ones that can sit active but unused.

6 Ways to reduce costs in Snowflake

1. Change your warehouse size

An easy solution to reduce Snowflake costs is to reduce the size of your virtual warehouse. We often think larger is better and more efficient in terms of compute, but this isn’t always the case. A lot of times we can reduce our warehouse size without suffering from performance issues. However, this isn’t always the case. You really need to play around and see what works for you.

Each warehouse size has a different number of available nodes. A query will utilize all of the nodes available within the warehouse to properly execute itself. However, if the query is small, it may not even use all of the available nodes. If it’s not using all of the available nodes, then you are paying for unused resources.

Again, you really don’t know which size will work best for you until you try. I would start with a small or medium size warehouse, unless you are querying large files, and experiment from there. Let your data pipeline run a few days with one size and keep track of performance and cost. Then, switch the size and measure the results. After you’ve tested a few different sizes, you’ll be able to see which optimizes both cost and performance for your particular use case.

In order to change the size of your virtual Snowflake warehouse, you can use the following command:

ALTER WAREHOUSE madison_test SET WAREHOUSE_SIZE= SMALL;

One important thing to remember with warehouses is that increasing the size of your warehouse does not always result in an increase in performance. A lot of times engineers will increase their warehouse size in hopes of their data pipeline running faster, but this can backfire in terms of cost and compute.

I often recommend creating multiple Snowflake warehouses, one for each function of your data stack, so you can see which activities cost the most. For example, I create separate warehouses for data ingestion, transformation, and analysis. Separating you warehouses by function will allow you to hone in on which parts of your stack need to be optimized.

2. Decrease the number of warehouses running at the same time

As I mentioned earlier, you pay for the time your warehouses are running. If you have multiple processes running, such as transformation and analysis, you are paying for the cost of two warehouses running at once. Despite what I mentioned earlier about creating different warehouses for different processes, you can lower your Snowflake costs by bringing this number down. However, you won’t know which process is utilizing the most compute power, making it a little fuzzy about which parts can be optimized.

This all just depends which is best for you. I recommend trying both options and seeing how much the cost savings is and then assessing whether that is worth not knowing which processes are running up costs. To successfully condense your warehouses, you first want to think about which processes run at the same time.

For me, data ingestion and transformation run hand in hand with my automated data pipeline. I also use the transformation warehouse to test dbt development models during the day, at the same time the data analyst is performing analyses. Instead of having three separate data warehouses, it may benefit me cost-wise to only have one to two warehouses for data pipeline activities and another warehouse for development.

Integrate with 700+ apps using Airbyte

Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.

Try it free Talk to sales

About the Author

Madison Schott

Madison Schott is an Analytics Engineer, author of The ABCs of Analytics Engineering book. Madison blogs about the modern data stack on Medium and writes a newsletter on Substack.

Integrate with 700+ apps using Airbyte

Try Airbyte for free

6 ways to reduce Snowflake costs

Integrate with 700+ apps using Airbyte

Integrate with 700+ apps using Airbyte

Related posts