Warehouses and Lakes
Finance & Ops Analytics

How to load data from Stripe to Snowflake destination

Learn how to use Airbyte to synchronize your Stripe data into Snowflake destination within minutes.

TL;DR

This can be done by building a data pipeline manually, usually a Python script (you can leverage a tool as Apache Airflow for this). This process can take more than a full week of development. Or it can be done in minutes on Airbyte in three easy steps:

  1. set up Stripe as a source connector (using Auth, or usually an API key)
  2. set up Snowflake destination as a destination connector
  3. define which data you want to transfer and how frequently

You can choose to self-host the pipeline using Airbyte Open Source or have it managed for you with Airbyte Cloud.

This tutorial’s purpose is to show you how.

What is Stripe

Stripe is a technology company focused on helping businesses of all sizes accept web and mobile payments. Stripe software is intended to build a solid economic infrastructure for the internet at global scale. Well-known companies like Salesforce and Facebook accept online payments through Stripe software. Stripe’s innovative applications combined with their solid economic infrastructure support modern business models like crowdfunding and marketplaces. Stripe continues to innovate, partnering with tech-dominant enterprises such as Apple, Google, and Facebook to launch new capabilities.

What is Snowflake destination

A cloud data platform, Snowflake Data Cloud provides a warehouse-as-a-service built specifically for the cloud. The Snowflake platform is designed to empower many types of data workloads, and offers secure, immediate, governed access to a comprehensive network of data. Snowflake’s innovative technology goes above the capabilities of the ordinary database, supplying users all the functionality of database storage, query processing, and cloud services in one package.

Prerequisites

  1. A Stripe account to transfer your customer data automatically from.
  2. A Snowflake destination account.
  3. An active Airbyte Cloud account, or you can also choose to use Airbyte Open Source locally. You can follow the instructions to set up Airbyte on your system using docker-compose.

Airbyte is an open-source data integration platform that consolidates and streamlines the process of extracting and loading data from multiple data sources to data warehouses. It offers pre-built connectors, including Stripe and Snowflake destination, for seamless data migration.

When using Airbyte to move data from Stripe to Snowflake destination, it extracts data from Stripe using the source connector, converts it into a format Snowflake destination can ingest using the provided schema, and then loads it into Snowflake destination via the destination connector. This allows businesses to leverage their Stripe data for advanced analytics and insights within Snowflake destination, simplifying the ETL process and saving significant time and resources.

Step 1: Set up Stripe as a source connector

1. First, you need to create a Stripe account and log in to your dashboard.

2. Navigate to the Developers section and click on API keys.

3. Copy the Secret Key and Publishable Key.

4. Go to Airbyte and click on Sources in the left-hand menu.

5. Click on the Stripe source connector and then click on Create New Connection.

6. Enter a name for your connection and paste the Secret Key and Publishable Key in the respective fields.

7. Click on Test Connection to ensure that the credentials are correct.

8. If the test is successful, click on Save to save the connection.

9. You can now configure the sync settings for your Stripe source connector, such as selecting the data you want to sync and setting up a schedule for the sync to run automatically.

10. Once you have configured the sync settings, click on Save and Run to start syncing your Stripe data with Airbyte.

Step 2: Set up Snowflake destination as a destination connector

1. First, navigate to the Airbyte website and log in to your account.

2. Once you are logged in, click on the "Destinations" tab on the left-hand side of the screen.

3. Scroll down until you find the Snowflake Data Cloud destination connector and click on it.

4. You will be prompted to enter your Snowflake account information, including your account name, username, and password.

5. After entering your account information, click on the "Test" button to ensure that the connection is successful.

6. If the test is successful, click on the "Save" button to save your Snowflake Data Cloud destination connector settings.

7. You can now use the Snowflake Data Cloud destination connector to transfer data from your Airbyte sources to your Snowflake account.

8. To set up a data transfer, navigate to the "Sources" tab on the left-hand side of the screen and select the source you want to transfer data from.

9. Click on the "Create New Connection" button and select the Snowflake Data Cloud destination connector as your destination.

10. Follow the prompts to set up your data transfer, including selecting the tables or data sources you want to transfer and setting up any necessary transformations or mappings.

11. Once you have set up your data transfer, click on the "Run" button to start the transfer process.

Should you build or buy your data pipelines?

Download our free guide and discover the best approach for your needs, whether it's building your ELT solution in-house or opting for Airbyte Open Source or Airbyte Cloud.

Download now

Step 3: Set up a connection to sync your Stripe data to Snowflake destination

Once you've successfully connected Stripe as a data source and Snowflake destination as a destination in Airbyte, you can set up a data pipeline between them with the following steps:

  1. Create a new connection: On the Airbyte dashboard, navigate to the 'Connections' tab and click the '+ New Connection' button.
  2. Choose your source: Select Stripe from the dropdown list of your configured sources.
  3. Select your destination: Choose Snowflake destination from the dropdown list of your configured destinations.
  4. Configure your sync: Define the frequency of your data syncs based on your business needs. Airbyte allows both manual and automatic scheduling for your data refreshes.
  5. Select the data to sync: Choose the specific Stripe objects you want to import data from towards Snowflake destination. You can sync all data or select specific tables and fields.
  6. Select the sync mode for your streams: Choose between full refreshes or incremental syncs (with deduplication if you want), and this for all streams or at the stream level. Incremental is only available for streams that have a primary cursor.
  7. Test your connection: Click the 'Test Connection' button to make sure that your setup works. If the connection test is successful, save your configuration.
  8. Start the sync: If the test passes, click 'Set Up Connection'. Airbyte will start moving data from Stripe to Snowflake destination according to your settings.

Remember, Airbyte keeps your data in sync at the frequency you determine, ensuring your Snowflake destination data warehouse is always up-to-date with your Stripe data.

Use Cases to transfer your Stripe data to Snowflake destination

Integrating data from Stripe to Snowflake destination provides several benefits. Here are a few use cases:

  1. Advanced Analytics: Snowflake destination’s powerful data processing capabilities enable you to perform complex queries and data analysis on your Stripe data, extracting insights that wouldn't be possible within Stripe alone.
  2. Data Consolidation: If you're using multiple other sources along with Stripe, syncing to Snowflake destination allows you to centralize your data for a holistic view of your operations, and to set up a change data capture process so you never have any discrepancies in your data again.
  3. Historical Data Analysis: Stripe has limits on historical data. Syncing data to Snowflake destination allows for long-term data retention and analysis of historical trends over time.
  4. Data Security and Compliance: Snowflake destination provides robust data security features. Syncing Stripe data to Snowflake destination ensures your data is secured and allows for advanced data governance and compliance management.
  5. Scalability: Snowflake destination can handle large volumes of data without affecting performance, providing an ideal solution for growing businesses with expanding Stripe data.
  6. Data Science and Machine Learning: By having Stripe data in Snowflake destination, you can apply machine learning models to your data for predictive analytics, customer segmentation, and more.
  7. Reporting and Visualization: While Stripe provides reporting tools, data visualization tools like Tableau, PowerBI, Looker (Google Data Studio) can connect to Snowflake destination, providing more advanced business intelligence options. If you have a Stripe table that needs to be converted to a Snowflake destination table, Airbyte can do that automatically.

Wrapping Up

To summarize, this tutorial has shown you how to:

  1. Configure a Stripe account as an Airbyte data source connector.
  2. Configure Snowflake destination as a data destination connector.
  3. Create an Airbyte data pipeline that will automatically be moving data directly from Stripe to Snowflake destination after you set a schedule

With Airbyte, creating data pipelines take minutes, and the data integration possibilities are endless. Airbyte supports the largest catalog of API tools, databases, and files, among other sources. Airbyte's connectors are open-source, so you can add any custom objects to the connector, or even build a new connector from scratch without any local dev environment or any data engineer within 10 minutes with the no-code connector builder.

We look forward to seeing you make use of it! We invite you to join the conversation on our community Slack Channel, or sign up for our newsletter. You should also check out other Airbyte tutorials, and Airbyte’s content hub!

What should you do next?

Hope you enjoyed the reading. Here are the 3 ways we can help you in your data journey:

flag icon
Easily address your data movement needs with Airbyte Cloud
Take the first step towards extensible data movement infrastructure that will give a ton of time back to your data team. 
Get started with Airbyte for free
high five icon
Talk to a data infrastructure expert
Get a free consultation with an Airbyte expert to significantly improve your data movement infrastructure. 
Talk to sales
stars sparkling
Improve your data infrastructure knowledge
Subscribe to our monthly newsletter and get the community’s new enlightening content along with Airbyte’s progress in their mission to solve data integration once and for all.
Subscribe to newsletter

Millions of businesses use Stripe APIs to accept payments, send payouts, and manage their online transactions. Nevertheless, Stripe’s analytical capabilities are limited, offering only a small set of metrics such as gross volumes, net sales, new customers, successful payments, and so on.

To improve your Stripe analytics capabilities, you can use Stripe’s REST APIs to extract your Stripe transactions and replicate your Stripe data into a modern data warehouse architecture such as Snowflake. Snowflake is purpose-built for online analytical processing (OLAP), and once your data is in Snowflake tables it can be easily combined with data from other internal systems, which will provide you with enhanced insights into your business.

One challenge that you may come across is that extracting your Stripe data may require skilled Stripe developers and extensive consultations to Stripe's developer documentation. Furthermore, once you get the data out of Stripe, that is only half the battle – you still need to drive your data into Snowflake! Luckily Airbyte makes this easy by providing a Stripe source connector and a Snowflake destination connector. Airbyte data connectors make it easy to build a data pipeline to move your data from Stripe to Snowflake without the need for extensive consultations to the Stripe developer portal or to Stripe’s developer docs.

Prerequisites

  1. Register for Airbyte Cloud or deploy Airbyte Open-Source.  This tutorial is presented using Airbyte Cloud, but most of it should also work with the Open-Source version of Airbyte.
  2. An active Stripe and Snowflake account.

Step 1: Set up your Stripe account

You will need an active Stripe account to follow this tutorial. Airbyte requires your Stripe Account ID and Secret Key. You can find the Account ID by visiting your account settings.

To find the Secret key, visit the API keys page. You can find more about managing API keys in Stripe developer documentation.

Step 2: Set up your Snowflake account

If you don’t already have a Snowflake account, you’ll need to pick a Snowflake edition and a cloud provider for your warehouse as part of the account creation process.

The Snowflake dashboard will appear after you create a username and password. The worksheet area will be where you’ll run scripts and SQL queries for creating, visualizing, and modifying resources.

For Airbyte to successfully sync data from Stripe, you need to create a data warehouse and a database. Luckily, Airbyte provides a nice ready-to-execute script that lets you do it in a few seconds. You can read more about setting up a Snowflake destination in Airbyte’s Snowflake docs.

The main steps in the following script are:

  1. Setting Airbyte variables to be used later in the script. Make sure to change the default values in the script according to your preferences. Also, make sure to use a secure password.
  2. Create a new sysadmin role
  3. Create a new user, AIRBYTE_USER, and grant them privileges
  4. Set up a warehouse AIRBYTE_WAREHOUSE.
  5. Create a database named AIRBYTE_DATABASE inside this warehouse


-- set variables (these need to be uppercase)
set airbyte_role = 'AIRBYTE_ROLE';
set airbyte_username = 'AIRBYTE_USER';
set airbyte_warehouse = 'AIRBYTE_WAREHOUSE';
set airbyte_database = 'AIRBYTE_DATABASE';
set airbyte_schema = 'AIRBYTE_SCHEMA';

-- set user password
set airbyte_password = 'password';

begin;

-- create Airbyte role
use role securityadmin;
create role if not exists identifier($airbyte_role);
grant role identifier($airbyte_role) to role SYSADMIN;

-- create Airbyte user
create user if not exists identifier($airbyte_username)
password = $airbyte_password
default_role = $airbyte_role
default_warehouse = $airbyte_warehouse;

grant role identifier($airbyte_role) to user identifier($airbyte_username);

-- change role to sysadmin for warehouse / database steps
use role sysadmin;

-- create Airbyte warehouse
create warehouse if not exists identifier($airbyte_warehouse)
warehouse_size = xsmall
warehouse_type = standard
auto_suspend = 60
auto_resume = true


initially_suspended = true;

-- create Airbyte database
create database if not exists identifier($airbyte_database);

-- grant Airbyte warehouse access
grant USAGE
on warehouse identifier($airbyte_warehouse)
to role identifier($airbyte_role);

-- grant Airbyte database access
grant OWNERSHIP
on database identifier($airbyte_database)
to role identifier($airbyte_role);

commit;

begin;

USE DATABASE identifier($airbyte_database);

-- create schema for Airbyte data
CREATE SCHEMA IF NOT EXISTS identifier($airbyte_schema);

commit;

begin;

-- grant Airbyte schema access
grant OWNERSHIP
on schema identifier($airbyte_schema)
to role identifier($airbyte_role);

commit;

You can copy and run this whole script in a new Snowflake worksheet. Make sure to select “All Queries” when executing the worksheet.

ℹ️  Note that you should rename all the values for database, warehouse, roles, and user variables in the Snowflake setup script.  For simplicity and testing purposes, you will be using default values. This example also uses the sysadmin role, and you should assign reduced permissions to the service account before deploying in production.

After you run the script, Snowflake will tell you whether the command was successfully executed. You can visit the warehouse section to confirm that there is a new warehouse with the name AIRBYTE_WAREHOUSE and the configuration that you defined in the script.

Now that you have configured Snowflake, you are ready to create an ELT pipeline from your Stripe source to your Snowflake destination.

Step 3: Set up Stripe as an Airbyte source

The Airbyte Stripe source connector can be used to sync the many different tables from your Stripe account.  Some examples of available Stripe tables are:

  1. Balance Transactions
  2. Bank Accounts
  3. Invoice
  4. Customers
  5. Charges
  6. Subscriptions

You will define a Stripe source which will be used by Airbyte to extract data from the Stripe REST API. Go to sources and create a new Stripe source from the dropdown.

You can choose whatever name you like for the Stripe source.  Use the Account ID and Secret Key that you found earlier in this tutorial. The replication date should be set to a date from before your Stripe account became operational.

Step 4: Set up Snowflake as your Airbyte destination

Go to the destinations tab in your Airbyte cloud dashboard and select Snowflake as a new destination.

You will need the following details:

  1. The Snowflake host, which is the Snowflake account identifier (E.g., abc123.sp-southeast-1.Snowflakecomputing.com)
  2. The role to be used for accessing the Snowflake warehouse/database
  3. Warehouse, which is the warehouse identifier
  4. The database name
  5. The schema name
  6. A username, which you created previously while setting up Snowflake
  7. The password of the above user.

Enter the data as follows:

For the purposes of this tutorial, do not set JDBC URL Params.

ℹ️  JDBC URL Params is an optional property may be used to pass custom key-value pairs for connecting to Snowflake.

The Data Staging Method defines where the data is stored (or staged) so that Snowflake can efficiently load it into tables. By default, Airbyte uses Snowflake’s Internal Stage to load data. You can also choose to load data from Amazon S3 or any other cloud host. You can read more about setting up a data loading method in Airbyte’s docs.

Create the destination. This may take a few seconds until Airbyte authentication takes place.

Step 5: Set up a Stripe to Snowflake Airbyte connection

The final step to create a Stripe data pipeline to Snowflake involves creating the connection between the source and destination. Go to the Connections tab to create a new connection. Select the Stripe source and Snowflake destination that you just created.

You will then see the following screen:

Choose a name for this connection, and choose your desired replication frequency. If you want to use the same naming as the source data in Stripe, then choose the “mirror source structure” option for your namespace configuration. Alternatively, you may define a prefix to be used in the destination.

Once you are satisfied with your configuration, press Set up connection and wait for the sync first to complete. Syncing all the namespaces can take a few minutes.

Once the sync has completed, you can look at the Databases tab in the Snowflake dashboard to confirm that the data has been replicated. You should see a page similar to the following:

You can now run advanced analytics and execute custom SQL queries on your Stripe data in Snowflake!

Wrapping up

Moving data from Stripe into Snowflake provides you with powerful analytics capabilities and allows you to see a full picture of your business. However, interacting with the Stripe REST API may require expert Stripe developers that know how to navigate Stripe developer docs. Furthermore, driving data into Snowflake may require understanding Snowflake's best practices and may need engineering resources and developer time. Alternatively you can use Airbyte to easily move your data from Stripe to Snowflake, as you have done in this tutorial.  Specifically, in this tutorial you have learned how to:

  1. Configure a Stripe account.
  2. Configure a Snowflake account by creating a new warehouse and database.
  3. Build an Airbyte connection between Stripe and Snowflake.
  4. Sync Stripe data to Snowflake.

If you have enjoyed this tutorial, you may be interested in other Airbyte tutorials, or in Airbyte’s blog. You can also join the conversation on our community Slack Channel, participate in discussions on Airbyte’s discourse, or sign up for our newsletter. Furthermore, if you are interested in Airbyte as a fully managed service, you can try Airbyte Cloud for free!

What should you do next?

Hope you enjoyed the reading. Here are the 3 ways we can help you in your data journey:

flag icon
Easily address your data movement needs with Airbyte Cloud
Take the first step towards extensible data movement infrastructure that will give a ton of time back to your data team. 
Get started with Airbyte for free
high five icon
Talk to a data infrastructure expert
Get a free consultation with an Airbyte expert to significantly improve your data movement infrastructure. 
Talk to sales
stars sparkling
Improve your data infrastructure knowledge
Subscribe to our monthly newsletter and get the community’s new enlightening content along with Airbyte’s progress in their mission to solve data integration once and for all.
Subscribe to newsletter

Should you build or buy your data pipelines?

Download our free guide and discover the best approach for your needs, whether it's building your ELT solution in-house or opting for Airbyte Open Source or Airbyte Cloud.

Download now

Frequently Asked Questions

What data can you extract from Stripe?

Stripe's API provides access to a wide range of data related to payment processing and management. The following are the categories of data that can be accessed through Stripe's API:  

1. Payment data: This includes information about payments made through Stripe, such as the amount, currency, and status of the payment.  

2. Customer data: This includes information about customers who have made payments through Stripe, such as their name, email address, and payment history.  

3. Subscription data: This includes information about subscriptions made through Stripe, such as the subscription plan, billing cycle, and status of the subscription.  

4. Dispute data: This includes information about disputes raised by customers, such as the reason for the dispute and the status of the dispute resolution process.  

5. Balance data: This includes information about the balance of the Stripe account, such as the available balance, pending balance, and currency.  

6. Transfer data: This includes information about transfers made from the Stripe account to a bank account, such as the amount, currency, and status of the transfer.  

7. Refund data: This includes information about refunds made through Stripe, such as the amount, currency, and status of the refund.  

Overall, Stripe's API provides access to a comprehensive set of data related to payment processing and management, enabling businesses to effectively manage their payment operations.

What data can you transfer to Snowflake destination?

You can transfer a wide variety of data to Snowflake destination. This usually includes structured, semi-structured, and unstructured data like transaction records, log files, JSON data, CSV files, and more, allowing robust, scalable data integration and analysis.

What are top ETL tools to transfer data from Stripe to Snowflake destination?

The most prominent ETL tools to transfer data from Stripe to Snowflake destination include:

  • Airbyte
  • Fivetran
  • Stitch
  • Matillion
  • Talend Data Integration

These tools help in extracting data from Stripe and various sources (APIs, databases, and more), transforming it efficiently, and loading it into Snowflake destination and other databases, data warehouses and data lakes, enhancing data management capabilities.