How to load data from Microsoft SQL Server (MSSQL) to Snowflake destination

Learn how to use Airbyte to synchronize your Microsoft SQL Server (MSSQL) data into Snowflake destination within minutes.

Trusted by data-driven companies

How Airbyte Works

About the source and destination

Microsoft SQL Server (MSSQL)

Microsoft SQL Server Consultants help companies choose the best business software solutions for their needs. Microsoft SQL Server Consultants help businesses resolve questions and issues, provide businesses with reliable information resources, and, ultimately, make better decisions on the software most appropriate for their unique needs. Consultants are available to help on call and can connect remotely to businesses’ computers to upgrade outdated editions of SQL servers to bring functions up to date for improved productivity.

Snowflake destination

A cloud data platform, Snowflake Data Cloud provides a warehouse-as-a-service built specifically for the cloud. The Snowflake platform is designed to empower many types of data workloads, and offers secure, immediate, governed access to a comprehensive network of data. Snowflake’s innovative technology goes above the capabilities of the ordinary database, supplying users all the functionality of database storage, query processing, and cloud services in one package.

Old Automated Content

TL;DR

This can be done by building a data pipeline manually, usually a Python script (you can leverage a tool as Apache Airflow for this). This process can take more than a full week of development. Or it can be done in minutes on Airbyte in three easy steps:

  1. set up Microsoft SQL Server (MSSQL) as a source connector (using Auth, or usually an API key)
  2. set up Snowflake destination as a destination connector
  3. define which data you want to transfer and how frequently

You can choose to self-host the pipeline using Airbyte Open Source or have it managed for you with Airbyte Cloud.

This tutorial’s purpose is to show you how.

What is Microsoft SQL Server (MSSQL)

Microsoft SQL Server Consultants help companies choose the best business software solutions for their needs. Microsoft SQL Server Consultants help businesses resolve questions and issues, provide businesses with reliable information resources, and, ultimately, make better decisions on the software most appropriate for their unique needs. Consultants are available to help on call and can connect remotely to businesses’ computers to upgrade outdated editions of SQL servers to bring functions up to date for improved productivity.

What is Snowflake destination

A cloud data platform, Snowflake Data Cloud provides a warehouse-as-a-service built specifically for the cloud. The Snowflake platform is designed to empower many types of data workloads, and offers secure, immediate, governed access to a comprehensive network of data. Snowflake’s innovative technology goes above the capabilities of the ordinary database, supplying users all the functionality of database storage, query processing, and cloud services in one package.

Integrate Microsoft SQL Server (MSSQL) with Snowflake destination in minutes

Try for free now

Prerequisites

  1. A Microsoft SQL Server (MSSQL) account to transfer your customer data automatically from.
  2. A Snowflake destination account.
  3. An active Airbyte Cloud account, or you can also choose to use Airbyte Open Source locally. You can follow the instructions to set up Airbyte on your system using docker-compose.

Airbyte is an open-source data integration platform that consolidates and streamlines the process of extracting and loading data from multiple data sources to data warehouses. It offers pre-built connectors, including Microsoft SQL Server (MSSQL) and Snowflake destination, for seamless data migration.

When using Airbyte to move data from Microsoft SQL Server (MSSQL) to Snowflake destination, it extracts data from Microsoft SQL Server (MSSQL) using the source connector, converts it into a format Snowflake destination can ingest using the provided schema, and then loads it into Snowflake destination via the destination connector. This allows businesses to leverage their Microsoft SQL Server (MSSQL) data for advanced analytics and insights within Snowflake destination, simplifying the ETL process and saving significant time and resources.

Step 1: Set up Microsoft SQL Server (MSSQL) as a source connector

1. Open the Airbyte platform and navigate to the "Sources" tab on the left-hand side of the screen.

2. Click on the "Add Source" button and select "MSSQL - SQL Server" from the list of available connectors.

3. Enter a name for the connector and click on the "Next" button.

4. Enter the required credentials for your MSSQL - SQL Server database, including the server name, port number, database name, username, and password.

5. Test the connection to ensure that the credentials are correct and the connection is successful.

6. Select the tables or views that you want to replicate from the MSSQL - SQL Server database.

7. Choose the replication mode that you want to use, either full or incremental.

8. Configure any additional settings, such as the replication frequency and the maximum number of rows to replicate.

9. Click on the "Create Source" button to save the configuration and start the replication process.

10. Monitor the replication process and troubleshoot any issues that may arise using the Airbyte platform's monitoring and logging features.

Step 2: Set up Snowflake destination as a destination connector

1. First, navigate to the Airbyte website and log in to your account.

2. Once you are logged in, click on the "Destinations" tab on the left-hand side of the screen.

3. Scroll down until you find the Snowflake Data Cloud destination connector and click on it.

4. You will be prompted to enter your Snowflake account information, including your account name, username, and password.

5. After entering your account information, click on the "Test" button to ensure that the connection is successful.

6. If the test is successful, click on the "Save" button to save your Snowflake Data Cloud destination connector settings.

7. You can now use the Snowflake Data Cloud destination connector to transfer data from your Airbyte sources to your Snowflake account.

8. To set up a data transfer, navigate to the "Sources" tab on the left-hand side of the screen and select the source you want to transfer data from.

9. Click on the "Create New Connection" button and select the Snowflake Data Cloud destination connector as your destination.

10. Follow the prompts to set up your data transfer, including selecting the tables or data sources you want to transfer and setting up any necessary transformations or mappings.

11. Once you have set up your data transfer, click on the "Run" button to start the transfer process.

Step 3: Set up a connection to sync your Microsoft SQL Server (MSSQL) data to Snowflake destination

Once you've successfully connected Microsoft SQL Server (MSSQL) as a data source and Snowflake destination as a destination in Airbyte, you can set up a data pipeline between them with the following steps:

  1. Create a new connection: On the Airbyte dashboard, navigate to the 'Connections' tab and click the '+ New Connection' button.
  2. Choose your source: Select Microsoft SQL Server (MSSQL) from the dropdown list of your configured sources.
  3. Select your destination: Choose Snowflake destination from the dropdown list of your configured destinations.
  4. Configure your sync: Define the frequency of your data syncs based on your business needs. Airbyte allows both manual and automatic scheduling for your data refreshes.
  5. Select the data to sync: Choose the specific Microsoft SQL Server (MSSQL) objects you want to import data from towards Snowflake destination. You can sync all data or select specific tables and fields.
  6. Select the sync mode for your streams: Choose between full refreshes or incremental syncs (with deduplication if you want), and this for all streams or at the stream level. Incremental is only available for streams that have a primary cursor.
  7. Test your connection: Click the 'Test Connection' button to make sure that your setup works. If the connection test is successful, save your configuration.
  8. Start the sync: If the test passes, click 'Set Up Connection'. Airbyte will start moving data from Microsoft SQL Server (MSSQL) to Snowflake destination according to your settings.

Remember, Airbyte keeps your data in sync at the frequency you determine, ensuring your Snowflake destination data warehouse is always up-to-date with your Microsoft SQL Server (MSSQL) data.

Use Cases to transfer your Microsoft SQL Server (MSSQL) data to Snowflake destination

Integrating data from Microsoft SQL Server (MSSQL) to Snowflake destination provides several benefits. Here are a few use cases:

  1. Advanced Analytics: Snowflake destination’s powerful data processing capabilities enable you to perform complex queries and data analysis on your Microsoft SQL Server (MSSQL) data, extracting insights that wouldn't be possible within Microsoft SQL Server (MSSQL) alone.
  2. Data Consolidation: If you're using multiple other sources along with Microsoft SQL Server (MSSQL), syncing to Snowflake destination allows you to centralize your data for a holistic view of your operations, and to set up a change data capture process so you never have any discrepancies in your data again.
  3. Historical Data Analysis: Microsoft SQL Server (MSSQL) has limits on historical data. Syncing data to Snowflake destination allows for long-term data retention and analysis of historical trends over time.
  4. Data Security and Compliance: Snowflake destination provides robust data security features. Syncing Microsoft SQL Server (MSSQL) data to Snowflake destination ensures your data is secured and allows for advanced data governance and compliance management.
  5. Scalability: Snowflake destination can handle large volumes of data without affecting performance, providing an ideal solution for growing businesses with expanding Microsoft SQL Server (MSSQL) data.
  6. Data Science and Machine Learning: By having Microsoft SQL Server (MSSQL) data in Snowflake destination, you can apply machine learning models to your data for predictive analytics, customer segmentation, and more.
  7. Reporting and Visualization: While Microsoft SQL Server (MSSQL) provides reporting tools, data visualization tools like Tableau, PowerBI, Looker (Google Data Studio) can connect to Snowflake destination, providing more advanced business intelligence options. If you have a Microsoft SQL Server (MSSQL) table that needs to be converted to a Snowflake destination table, Airbyte can do that automatically.

Wrapping Up

To summarize, this tutorial has shown you how to:

  1. Configure a Microsoft SQL Server (MSSQL) account as an Airbyte data source connector.
  2. Configure Snowflake destination as a data destination connector.
  3. Create an Airbyte data pipeline that will automatically be moving data directly from Microsoft SQL Server (MSSQL) to Snowflake destination after you set a schedule

With Airbyte, creating data pipelines take minutes, and the data integration possibilities are endless. Airbyte supports the largest catalog of API tools, databases, and files, among other sources. Airbyte's connectors are open-source, so you can add any custom objects to the connector, or even build a new connector from scratch without any local dev environment or any data engineer within 10 minutes with the no-code connector builder.

We look forward to seeing you make use of it! We invite you to join the conversation on our community Slack Channel, or sign up for our newsletter. You should also check out other Airbyte tutorials, and Airbyte’s content hub!

What should you do next?

Hope you enjoyed the reading. Here are the 3 ways we can help you in your data journey:

flag icon
Easily address your data movement needs with Airbyte Cloud
Take the first step towards extensible data movement infrastructure that will give a ton of time back to your data team. 
Get started with Airbyte for free
high five icon
Talk to a data infrastructure expert
Get a free consultation with an Airbyte expert to significantly improve your data movement infrastructure. 
Talk to sales
stars sparkling
Improve your data infrastructure knowledge
Subscribe to our monthly newsletter and get the community’s new enlightening content along with Airbyte’s progress in their mission to solve data integration once and for all.
Subscribe to newsletter

Sync with Airbyte

Sync Manually

What sets Airbyte Apart

Modern GenAI Workflows

Streamline AI workflows with Airbyte: load unstructured data into vector stores like Pinecone, Weaviate, and Milvus. Supports RAG transformations with LangChain chunking and embeddings from OpenAI, Cohere, etc., all in one operation.

Move Large Volumes, Fast

Quickly get up and running with a 5-minute setup that supports both incremental and full refreshes, for databases of any size.

An Extensible Open-Source Standard

More than 1,000 developers contribute to Airbyte’s connectors, different interfaces (UI, API, Terraform Provider, Python Library), and integrations with the rest of the stack. Airbyte’s Connector Builder lets you edit or add new connectors in minutes.

Full Control & Security

Airbyte secures your data with cloud-hosted, self-hosted or hybrid deployment options. Single Sign-On (SSO) and Role-Based Access Control (RBAC) ensure only authorized users have access with the right permissions. Airbyte acts as a HIPAA conduit and supports compliance with CCPA, GDPR, and SOC2.

Fully Featured & Integrated

Airbyte automates schema evolution for seamless data flow, and utilizes efficient Change Data Capture (CDC) for real-time updates. Select only the columns you need, and leverage our dbt integration for powerful data transformations.

Enterprise Support with SLAs

Airbyte Self-Managed Enterprise comes with dedicated support and guaranteed service level agreements (SLAs), ensuring that your data movement infrastructure remains reliable and performant, and expert assistance is available when needed.
Warehouses and Lakes
Databases

How to load data from Microsoft SQL Server (MSSQL) to Snowflake destination

Learn how to use Airbyte to synchronize your Microsoft SQL Server (MSSQL) data into Snowflake destination within minutes.

TL;DR

This can be done by building a data pipeline manually, usually a Python script (you can leverage a tool as Apache Airflow for this). This process can take more than a full week of development. Or it can be done in minutes on Airbyte in three easy steps:

  1. set up Microsoft SQL Server (MSSQL) as a source connector (using Auth, or usually an API key)
  2. set up Snowflake destination as a destination connector
  3. define which data you want to transfer and how frequently

You can choose to self-host the pipeline using Airbyte Open Source or have it managed for you with Airbyte Cloud.

This tutorial’s purpose is to show you how.

What is Microsoft SQL Server (MSSQL)

Microsoft SQL Server Consultants help companies choose the best business software solutions for their needs. Microsoft SQL Server Consultants help businesses resolve questions and issues, provide businesses with reliable information resources, and, ultimately, make better decisions on the software most appropriate for their unique needs. Consultants are available to help on call and can connect remotely to businesses’ computers to upgrade outdated editions of SQL servers to bring functions up to date for improved productivity.

What is Snowflake destination

A cloud data platform, Snowflake Data Cloud provides a warehouse-as-a-service built specifically for the cloud. The Snowflake platform is designed to empower many types of data workloads, and offers secure, immediate, governed access to a comprehensive network of data. Snowflake’s innovative technology goes above the capabilities of the ordinary database, supplying users all the functionality of database storage, query processing, and cloud services in one package.

Integrate Microsoft SQL Server (MSSQL) with Snowflake destination in minutes

Try for free now

Prerequisites

  1. A Microsoft SQL Server (MSSQL) account to transfer your customer data automatically from.
  2. A Snowflake destination account.
  3. An active Airbyte Cloud account, or you can also choose to use Airbyte Open Source locally. You can follow the instructions to set up Airbyte on your system using docker-compose.

Airbyte is an open-source data integration platform that consolidates and streamlines the process of extracting and loading data from multiple data sources to data warehouses. It offers pre-built connectors, including Microsoft SQL Server (MSSQL) and Snowflake destination, for seamless data migration.

When using Airbyte to move data from Microsoft SQL Server (MSSQL) to Snowflake destination, it extracts data from Microsoft SQL Server (MSSQL) using the source connector, converts it into a format Snowflake destination can ingest using the provided schema, and then loads it into Snowflake destination via the destination connector. This allows businesses to leverage their Microsoft SQL Server (MSSQL) data for advanced analytics and insights within Snowflake destination, simplifying the ETL process and saving significant time and resources.

Step 1: Set up Microsoft SQL Server (MSSQL) as a source connector

1. Open the Airbyte platform and navigate to the "Sources" tab on the left-hand side of the screen.

2. Click on the "Add Source" button and select "MSSQL - SQL Server" from the list of available connectors.

3. Enter a name for the connector and click on the "Next" button.

4. Enter the required credentials for your MSSQL - SQL Server database, including the server name, port number, database name, username, and password.

5. Test the connection to ensure that the credentials are correct and the connection is successful.

6. Select the tables or views that you want to replicate from the MSSQL - SQL Server database.

7. Choose the replication mode that you want to use, either full or incremental.

8. Configure any additional settings, such as the replication frequency and the maximum number of rows to replicate.

9. Click on the "Create Source" button to save the configuration and start the replication process.

10. Monitor the replication process and troubleshoot any issues that may arise using the Airbyte platform's monitoring and logging features.

Step 2: Set up Snowflake destination as a destination connector

1. First, navigate to the Airbyte website and log in to your account.

2. Once you are logged in, click on the "Destinations" tab on the left-hand side of the screen.

3. Scroll down until you find the Snowflake Data Cloud destination connector and click on it.

4. You will be prompted to enter your Snowflake account information, including your account name, username, and password.

5. After entering your account information, click on the "Test" button to ensure that the connection is successful.

6. If the test is successful, click on the "Save" button to save your Snowflake Data Cloud destination connector settings.

7. You can now use the Snowflake Data Cloud destination connector to transfer data from your Airbyte sources to your Snowflake account.

8. To set up a data transfer, navigate to the "Sources" tab on the left-hand side of the screen and select the source you want to transfer data from.

9. Click on the "Create New Connection" button and select the Snowflake Data Cloud destination connector as your destination.

10. Follow the prompts to set up your data transfer, including selecting the tables or data sources you want to transfer and setting up any necessary transformations or mappings.

11. Once you have set up your data transfer, click on the "Run" button to start the transfer process.

Step 3: Set up a connection to sync your Microsoft SQL Server (MSSQL) data to Snowflake destination

Once you've successfully connected Microsoft SQL Server (MSSQL) as a data source and Snowflake destination as a destination in Airbyte, you can set up a data pipeline between them with the following steps:

  1. Create a new connection: On the Airbyte dashboard, navigate to the 'Connections' tab and click the '+ New Connection' button.
  2. Choose your source: Select Microsoft SQL Server (MSSQL) from the dropdown list of your configured sources.
  3. Select your destination: Choose Snowflake destination from the dropdown list of your configured destinations.
  4. Configure your sync: Define the frequency of your data syncs based on your business needs. Airbyte allows both manual and automatic scheduling for your data refreshes.
  5. Select the data to sync: Choose the specific Microsoft SQL Server (MSSQL) objects you want to import data from towards Snowflake destination. You can sync all data or select specific tables and fields.
  6. Select the sync mode for your streams: Choose between full refreshes or incremental syncs (with deduplication if you want), and this for all streams or at the stream level. Incremental is only available for streams that have a primary cursor.
  7. Test your connection: Click the 'Test Connection' button to make sure that your setup works. If the connection test is successful, save your configuration.
  8. Start the sync: If the test passes, click 'Set Up Connection'. Airbyte will start moving data from Microsoft SQL Server (MSSQL) to Snowflake destination according to your settings.

Remember, Airbyte keeps your data in sync at the frequency you determine, ensuring your Snowflake destination data warehouse is always up-to-date with your Microsoft SQL Server (MSSQL) data.

Use Cases to transfer your Microsoft SQL Server (MSSQL) data to Snowflake destination

Integrating data from Microsoft SQL Server (MSSQL) to Snowflake destination provides several benefits. Here are a few use cases:

  1. Advanced Analytics: Snowflake destination’s powerful data processing capabilities enable you to perform complex queries and data analysis on your Microsoft SQL Server (MSSQL) data, extracting insights that wouldn't be possible within Microsoft SQL Server (MSSQL) alone.
  2. Data Consolidation: If you're using multiple other sources along with Microsoft SQL Server (MSSQL), syncing to Snowflake destination allows you to centralize your data for a holistic view of your operations, and to set up a change data capture process so you never have any discrepancies in your data again.
  3. Historical Data Analysis: Microsoft SQL Server (MSSQL) has limits on historical data. Syncing data to Snowflake destination allows for long-term data retention and analysis of historical trends over time.
  4. Data Security and Compliance: Snowflake destination provides robust data security features. Syncing Microsoft SQL Server (MSSQL) data to Snowflake destination ensures your data is secured and allows for advanced data governance and compliance management.
  5. Scalability: Snowflake destination can handle large volumes of data without affecting performance, providing an ideal solution for growing businesses with expanding Microsoft SQL Server (MSSQL) data.
  6. Data Science and Machine Learning: By having Microsoft SQL Server (MSSQL) data in Snowflake destination, you can apply machine learning models to your data for predictive analytics, customer segmentation, and more.
  7. Reporting and Visualization: While Microsoft SQL Server (MSSQL) provides reporting tools, data visualization tools like Tableau, PowerBI, Looker (Google Data Studio) can connect to Snowflake destination, providing more advanced business intelligence options. If you have a Microsoft SQL Server (MSSQL) table that needs to be converted to a Snowflake destination table, Airbyte can do that automatically.

Wrapping Up

To summarize, this tutorial has shown you how to:

  1. Configure a Microsoft SQL Server (MSSQL) account as an Airbyte data source connector.
  2. Configure Snowflake destination as a data destination connector.
  3. Create an Airbyte data pipeline that will automatically be moving data directly from Microsoft SQL Server (MSSQL) to Snowflake destination after you set a schedule

With Airbyte, creating data pipelines take minutes, and the data integration possibilities are endless. Airbyte supports the largest catalog of API tools, databases, and files, among other sources. Airbyte's connectors are open-source, so you can add any custom objects to the connector, or even build a new connector from scratch without any local dev environment or any data engineer within 10 minutes with the no-code connector builder.

We look forward to seeing you make use of it! We invite you to join the conversation on our community Slack Channel, or sign up for our newsletter. You should also check out other Airbyte tutorials, and Airbyte’s content hub!

What should you do next?

Hope you enjoyed the reading. Here are the 3 ways we can help you in your data journey:

flag icon
Easily address your data movement needs with Airbyte Cloud
Take the first step towards extensible data movement infrastructure that will give a ton of time back to your data team. 
Get started with Airbyte for free
high five icon
Talk to a data infrastructure expert
Get a free consultation with an Airbyte expert to significantly improve your data movement infrastructure. 
Talk to sales
stars sparkling
Improve your data infrastructure knowledge
Subscribe to our monthly newsletter and get the community’s new enlightening content along with Airbyte’s progress in their mission to solve data integration once and for all.
Subscribe to newsletter

SQL Server to Snowflake replication is crucial for many organizations looking to leverage the power of cloud data warehousing while maintaining their existing on-premises databases. This article explores two methods to achieve this replication: using Airbyte, an open-source data integration platform, and a manual approach involving data export and import using Snowflake's COPY command.

What is SQL Server?

SQL Server is a relational database management system developed by Microsoft. It's primarily used for storing and retrieving data as requested by other software applications, which may run on the same computer or across a network. SQL Server is widely used in enterprise environments for transactional systems, data warehousing, and business intelligence applications.

What is Snowflake?

Snowflake is a cloud-based data warehousing platform that provides a single integrated solution for data storage, processing, and analytics. It's designed to be highly scalable, allowing organizations to store and analyze massive amounts of structured and semi-structured data. Snowflake is particularly popular for its ability to separate compute and storage resources, enabling flexible scaling and cost-effective operations.

Benefits of moving data from Microsoft SQL Server to Snowflake

Moving data from MSSQL to Snowflake may be part of an overall data integration strategy, which will provide your organization with:

  • A unified view of data and a single source of truth – achieved by copying data from MSSQL Server and other operational systems into Snowflake.
  • Improved analytics capabilities – Snowflake is purpose built for running large analytics jobs.
  • The ability to transform data in a single location – moving data from multiple systems into Snowflake allows you to transform and join data from multiple disparate systems.
  • Improved security – limit the number of people that require access to your MSSQL Server, as they can analyze your MSSQL data in Snowflake.

In addition to the benefits listed above, Snowflake is designed for storing massive amounts of data. Therefore MSSQL replication into Snowflake may be used for backups, or for archiving historical MSSQL data as required for compliance or regulatory requirements.

{{COMPONENT_CTA}}

Methods to Move Data From Microsoft sql server to snowflake

  • Method 1: Connecting Microsoft sql server to snowflake using Airbyte
  • Method 2: Connecting Microsoft sql server to snowflake manually

Method 1: Connecting Microsoft sql server to snowflake using Airbyte

Prerequisites

In this tutorial Airbyte OSS will be used to replicate your Microsoft SQL Server data to Snowflake. You will therefore need the following prerequisites:

  1. Microsoft SQL Server.
  2. Airbyte OSS.
  3. Snowflake.

Step 1: Configuring Microsoft SQL Server

Depending on your operating system, you may use the Microsoft SQL Server docker image or you can install SQL Server on Windows by downloading the .exe installer file. In this example, we will set up SQL Server on macOS using Docker. First, download the latest version of the Microsoft SQL server image by running the following command.


docker pull mcr.microsoft.com/mssql/server

Once downloaded, start an instance by running the following command. First, choose a password by setting it in the highlighted section.


docker run --name airbyte-mssql -e "ACCEPT_EULA=Y" -e "SA_PASSWORD={YOUR_PASSWORD}" -e "MSSQL_AGENT_ENABLED=True" -p 1433:1433 -d mcr.microsoft.com/mssql/server:latest

You can also run T-SQL containers by connecting to the sqlcmd, which is a command-line shell for SQL Server. Again, use the same password configured in the previous step.


docker exec -it airbyte-mssql /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P {YOUR_PASSWORD}

Create a new database by running the following commands:


CREATE DATABASE Airbyte;
GO

You can verify that the database is created by running the following:


SELECT Name FROM sys.Databases;
GO

Which should respond with the following:

Run the following T-SQL Statements to create schema and tables that will be used as our sample data:


USE Airbyte;
GO

CREATE SCHEMA sales;
GO

CREATE TABLE sales.customers (
  customer_id INT IDENTITY (1, 1) PRIMARY KEY,
  first_name VARCHAR (255) NOT NULL,
  last_name VARCHAR (255) NOT NULL,
  phone VARCHAR (25),
  email VARCHAR (255) NOT NULL,
  street VARCHAR (255),
  city VARCHAR (50),
  state VARCHAR (25),
  zip_code VARCHAR (5)
);

CREATE TABLE sales.stores (
  store_id INT IDENTITY (1, 1) PRIMARY KEY,
  store_name VARCHAR (255) NOT NULL,
  phone VARCHAR (25),
  email VARCHAR (255),
  street VARCHAR (255),
  city VARCHAR (255),
  state VARCHAR (10),
  zip_code VARCHAR (5)
);

Add some rows to the customers table by running the following:



INSERT INTO sales.customers(first_name, last_name, phone, email, street, city, state, zip_code)
VALUES('Debra','Burks',NULL,'debra.burks@yahoo.com','9273 Thorne Ave. ','Orchard Park','NY',14127);
INSERT INTO sales.customers(first_name, last_name, phone, email, street, city, state, zip_code) VALUES('Kasha','Todd',NULL,'kasha.todd@yahoo.com','910 Vine Street ','Campbell','CA',95008);
INSERT INTO sales.customers(first_name, last_name, phone, email, street, city, state, zip_code) VALUES('Tameka','Fisher',NULL,'tameka.fisher@aol.com','769C Honey Creek St. ','Redondo Beach','CA',90278);
INSERT INTO sales.customers(first_name, last_name, phone, email, street, city, state, zip_code) VALUES('Daryl','Spence',NULL,'daryl.spence@aol.com','988 Pearl Lane ','Uniondale','NY',11553);
INSERT INTO sales.customers(first_name, last_name, phone, email, street, city, state, zip_code) VALUES('Charlotte','Rice','(916) 381-6003','charolette.rice@msn.com','107 River Dr. ','Sacramento','CA',95820);

GO

Add rows to the stores table by executing:



INSERT INTO sales.stores(store_name, phone, email, street, city, state, zip_code)
VALUES('Santa Cruz Bikes','(831) 476-4321','santacruz@bikes.shop','3700 Portola Drive', 'Santa Cruz','CA',95060),
    ('Baldwin Bikes','(516) 379-8888','baldwin@bikes.shop','4200 Chestnut Lane', 'Baldwin','NY',11432),
    ('Rowlett Bikes','(972) 530-5555','rowlett@bikes.shop','8000 Fairway Avenue', 'Rowlett','TX',75088);

GO

Step 2: Setting up your Snowflake Account

If you don’t already have a Snowflake account, you can create a trial account. When creating a Snowflake account, you’ll need to pick a Snowflake edition and a cloud provider as part of the account creation process.

Once your account is successfully created, you'll be redirected to the Snowflake dashboard. The worksheet area will be the primary place you’ll run scripts for creating and modifying resources. You will need to set up the destination database, user, role, and schema on Snowflake for the sync.

Airbyte provides a convenient script in the Snowflake destination connector documentation which you should copy into your Snowflake worksheet area. After you have copied the script into your Snowflake worksheet select ‘All queries’ and run the script by clicking on the run button as shown below.

ℹ️ Before running the script, be sure to change the airbyte_password variable to your preferred password value.

Step 3: Setting up Microsoft SQL Server as the Airbyte Source

Go to Airbyte and create a new source connection. Give the connection a name and select Microsoft SQL Server as the Source Type.

ℹ️ See the Microsoft SQL Server source connector documentation for additional information.

Enter the values for the host and port you configured when setting up your MSSQL Server docker container as shown below:

Step 4: Setting up Snowflake Destination in Airbyte

Set Snowflake as the destination, give the destination a name, and select Snowflake as the destination type.

ℹ️ See the Snowflake destination connector documentation for more information.

Enter the values for the fields based on the values set in the script in Step 2. For example, enter the URL you received by email for the host when signing up for Snowflake. If you updated the password in your script, enter the new password.

Step 5: Setting up the SQL Server to Snowflake Airbyte Connection

Once the source and destination connectors are configured, you can access your connection settings. You should be able to see the tables that are available in your Microsoft SQL Server as shown below.

Set the sync frequency and choose your Sync mode. In this example, the Full refresh | Append mode has been selected.

Step 6: Sync your data

Save the connection and select Sync now.

Once the Sync is complete, you can go to the Database section in the Snowflake UI to see the tables that have been copied. Snowflake should contain the normalized data in the same format as the SQL Server Table. The replica also includes the raw data in a separate set with the name _AIRBYTE_RAW_{TABLE_NAME}

You can view the structure of the table and the data types for each of the fields. Airbyte automatically maps the data types in the SQL Server tables to the corresponding data types in Snowflake.

Method 2: Connecting Microsoft sql server to snowflake manually

Moving data from Microsoft SQL Server to Snowflake without third-party connectors or integrations involves several steps, including exporting data from SQL Server, preparing the data for Snowflake, and importing the data into Snowflake. Below is a detailed guide to accomplish this task.

Step 1: Export Data from Microsoft SQL Server

1. Choose the Data to Export: Decide which tables or data you want to move from SQL Server to Snowflake.

2. Generate Scripts for Table Schema:

   - Connect to your SQL Server instance using SQL Server Management Studio (SSMS).

   - Right-click the database containing the data you want to export.

   - Navigate to Tasks > Generate Scripts.

   - Follow the wizard to select the specific tables and choose the schema only.

   - Save the scripts to a file.

3. Export Data to Flat Files:

   - In SSMS, right-click the database again and navigate to Tasks > Export Data.

   - Use the SQL Server Import and Export Wizard to export data.

   - Select the data source and the destination as a flat file format (CSV is commonly used).

   - Configure the flat file destination with the appropriate field terminators and encoding.

   - Run the package to export the data to the chosen location.

4. Validate the Exported Data: Ensure that the data has been exported correctly and completely by checking a few records in the flat file against the source database.

Step 2: Prepare Data for Snowflake

1. Modify the Table Schema Scripts: 

   - Edit the previously saved schema scripts to match Snowflake's syntax.

   - Adjust data types and remove any SQL Server-specific constructs that are not compatible with Snowflake.

   - Save the modified scripts.

2. Prepare the Flat Files:

   - If necessary, modify the CSV files to meet Snowflake's requirements (e.g., UTF-8 encoding).

   - Ensure that the files are accessible from a location that Snowflake can access, such as an Amazon S3 bucket or Azure Blob Storage.

3. Stage Files for Snowflake:

   - Upload the CSV files to a cloud storage location supported by Snowflake (Amazon S3, Google Cloud Storage, or Azure Blob Storage).

   - Verify that the files are successfully uploaded and accessible.

Step 3: Import Data into Snowflake

1. Set Up Snowflake Environment:

   - Log in to your Snowflake account.

   - Create a database and schema if they don't already exist.

   - Use the modified schema scripts to create the tables within Snowflake.

2. Create a File Format:

   - Create a file format in Snowflake that matches the format of your exported CSV files, including field delimiter, encoding, etc.

3. Create a Stage:

   - Create a stage in Snowflake that points to the location of the CSV files in the cloud storage.

   - Use the previously created file format in the stage definition.

4. Copy Data into Snowflake:

   - Use the `COPY INTO` command to load data from the staged files into the corresponding tables in Snowflake.

   - Verify the success of the data load by checking the number of rows loaded and looking for any errors.

5. Validate the Imported Data:

   - Run queries against the imported tables to ensure that the data has been loaded correctly.

   - Compare record counts and sample data between the source and destination.

6. Perform Post-Load Tasks:

   - Apply any necessary transformations or data cleanup in Snowflake.

   - Create additional indexes, views, or stored procedures as needed.

Step 4: Clean Up

1. Remove Temporary Files:

   - Once the data is verified, you can remove the CSV files from the cloud storage to avoid unnecessary storage costs.

2. Document the Process:

   - Document the steps taken, any issues encountered, and how they were resolved for future reference.

3. Automate the Process (Optional):

   - If this is a recurring task, consider automating the process using Snowflake's tasks or stored procedures, or SQL Server Agent jobs, to streamline future data transfers.

Remember, the complexity of the migration can vary significantly based on the size and complexity of the data, as well as specific requirements such as data transformation, data type mapping, and handling of large binary objects (LOBs). Always perform thorough testing to ensure data integrity throughout the migration process.

Which SQL Server to Snowflake replication method should you choose?

Ease of use and setup

Airbyte generally offers an easier setup process with its user-friendly interface and pre-built connectors. It abstracts much of the complexity involved in data replication.

Scalability and performance

Airbyte is designed to handle large-scale data replication efficiently and can automatically manage incremental updates. It's built to scale with your data needs. The manual method may struggle with very large datasets and requires custom scripting for efficient incremental updates, potentially leading to performance issues at scale.

Data consistency and integrity

Both methods can maintain data consistency, but Airbyte has built-in features for error handling and data validation.

Automation capabilities

Airbyte excels in automation, offering scheduled runs and monitoring features out of the box. The manual method requires additional tools or custom scripts to achieve similar levels of automation, making it more time-consuming to set up and maintain.

Learning curve and skill requirements

Airbyte has a shorter learning curve for teams already familiar with ETL tools. The manual method requires deeper knowledge of both SQL Server and Snowflake, as well as data export/import processes and potentially scripting languages.

Wrapping up

Both methods offer viable solutions for replicating data from SQL Server to Snowflake. Airbyte provides an automated, user-friendly approach with built-in scheduling and error handling, making it ideal for organizations seeking a streamlined, low-maintenance solution. The manual method, while more labor-intensive, offers greater control over the process and may be suitable for occasional transfers or smaller datasets.

Ultimately, the choice depends on your specific needs, technical expertise, and resources. Consider factors such as data volume, frequency of updates, and available manpower when deciding between these two approaches.

Ready to simplify your data replication process? Sign up for Airbyte today and experience seamless, automated data transfers between SQL Server and Snowflake.

If you have enjoyed this tutorial, you may be interested in other Airbyte tutorials, or in Airbyte’s blog. You can also join the conversation on our community Slack Channel, participate in discussions on Airbyte’s discourse, or sign up for our newsletter.

What should you do next?

Hope you enjoyed the reading. Here are the 3 ways we can help you in your data journey:

flag icon
Easily address your data movement needs with Airbyte Cloud
Take the first step towards extensible data movement infrastructure that will give a ton of time back to your data team. 
Get started with Airbyte for free
high five icon
Talk to a data infrastructure expert
Get a free consultation with an Airbyte expert to significantly improve your data movement infrastructure. 
Talk to sales
stars sparkling
Improve your data infrastructure knowledge
Subscribe to our monthly newsletter and get the community’s new enlightening content along with Airbyte’s progress in their mission to solve data integration once and for all.
Subscribe to newsletter

Frequently Asked Questions

What data can you extract from Microsoft SQL Server (MSSQL)?

MSSQL - SQL Server provides access to a wide range of data types, including:  

1. Relational data: This includes tables, views, and stored procedures that are used to store and manipulate data in a structured format.  

2. Non-relational data: This includes data that is not stored in a structured format, such as XML documents, JSON objects, and binary data.  

3. Spatial data: This includes data that is related to geographic locations, such as maps, coordinates, and spatial queries.  

4. Time-series data: This includes data that is related to time, such as timestamps, dates, and time intervals.  

5. Graph data: This includes data that is related to relationships between entities, such as social networks, supply chains, and organizational structures.  

6. Machine learning data: This includes data that is used for training and testing machine learning models, such as feature vectors, labels, and performance metrics.  

7. Streaming data: This includes data that is generated in real-time, such as sensor data, log files, and social media feeds.

What data can you transfer to Snowflake destination?

You can transfer a wide variety of data to Snowflake destination. This usually includes structured, semi-structured, and unstructured data like transaction records, log files, JSON data, CSV files, and more, allowing robust, scalable data integration and analysis.

What are top ETL tools to transfer data from Microsoft SQL Server (MSSQL) to Snowflake destination?

The most prominent ETL tools to transfer data from Microsoft SQL Server (MSSQL) to Snowflake destination include:

  • Airbyte
  • Fivetran
  • Stitch
  • Matillion
  • Talend Data Integration

These tools help in extracting data from Microsoft SQL Server (MSSQL) and various sources (APIs, databases, and more), transforming it efficiently, and loading it into Snowflake destination and other databases, data warehouses and data lakes, enhancing data management capabilities.

What should you do next?

Hope you enjoyed the reading. Here are the 3 ways we can help you in your data journey:

flag icon
Easily address your data movement needs with Airbyte Cloud
Take the first step towards extensible data movement infrastructure that will give a ton of time back to your data team. 
Get started with Airbyte for free
high five icon
Talk to a data infrastructure expert
Get a free consultation with an Airbyte expert to significantly improve your data movement infrastructure. 
Talk to sales
stars sparkling
Improve your data infrastructure knowledge
Subscribe to our monthly newsletter and get the community’s new enlightening content along with Airbyte’s progress in their mission to solve data integration once and for all.
Subscribe to newsletter