REST API to Snowflake: How to Move Your Data

Getting data from a REST API into Snowflake is the most common request in data engineering that has no single answer, because a REST API is not a system. It is a shape. What you build depends entirely on how the specific API handles authentication, pagination, and the question of what changed since last time.

One clarification first, because it costs people an afternoon. Airbyte has a connector named Public APIs, and it does not mean any public API. It syncs the catalogue at publicapis.org, a directory of free APIs, with two streams and no pagination. It is a useful toy and not what you want here. This guide covers the two real options for an arbitrary API and how to land the result in Snowflake.

REST API to Snowflake at a glance:

Why move data from a REST API to Snowflake?

Two situations account for most of these pipelines.

The first is a vendor or internal service with no maintained connector whose data belongs in your warehouse alongside everything else. The alternative is a scheduled script that somebody owns until they do not, and a declaratively defined connector removes that ownership problem because it lives in Airbyte with your other pipelines.

The second is completing a picture. Most analytical questions fail because one input is missing, and that input is often behind an API nobody has bothered to integrate. Snowflake is a good destination for this specifically because it handles semi-structured data natively, which matters more than usual when your source is an API.

What do you need before you start?

Most of this is reading the API's documentation carefully, which is the harder skill:

An authentication method you understand. Bearer token, API key in a header or query parameter, basic auth, or OAuth 2.0 with a refresh flow. Connector Builder supports these directly, and anything genuinely unusual is worth establishing now rather than halfway through.

The pagination scheme, named precisely. Page number, offset, or a cursor token returned in the response body or a Link header. Getting this wrong is the single most common reason a custom connector returns exactly one page and looks like it worked.

An answer on incremental filtering. Does any endpoint accept an updated_since parameter or an equivalent? If yes, configure a datetime cursor. If no, you are doing full refresh, and it is better to know that going in than to discover it when the vendor bills you for a daily full scan.

Snowflake objects and a role. A warehouse, database, schema and a role able to create tables in that schema. A dedicated role keeps credit consumption attributable, which matters when a full refresh pipeline runs more often than it should.

If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list on the Snowflake side before you begin.

How do you build a REST API to Snowflake pipeline in Airbyte?

Step 1: Check whether a connector already exists

Before building anything, search the catalogue for the service you are integrating. With 600+ connectors available, a maintained one covering your source is a real possibility, and a maintained connector handles the rate limits, schema changes and pagination quirks you would otherwise be discovering yourself. Only build when nothing fits.

Step 2: Explore with the HTTP Request source

Airbyte ships a generic HTTP Request source. Point it at a URL, give it credentials, and it retrieves the response, which is the fastest way to answer whether the data is even useful. It is deliberately unsophisticated: no stream modelling, no managed schema, and incremental behaviour is whatever you construct around it. Use it to explore, not to run a pipeline for two years.

Step 3: Define the connector in Connector Builder

Connector Builder is the low-code interface for defining a real source connector. You describe the API declaratively and Airbyte generates something that behaves like any catalogue connector, including in scheduling and state management. Configure authentication, then pagination, then your streams, pointing Builder at the record array inside each response since most APIs wrap results in an envelope. The testing panel runs live requests as you configure, so validate pagination against real responses before publishing.

Step 4: Configure Snowflake and connect

Click Destinations, then New Destination, and select Snowflake, following adding a destination. Then create the connection, select your streams and a sync mode for each, and set a frequency that respects whatever rate limit the API applies. On full refresh streams, that frequency is also a credit consumption decision.

How do nested API responses land in Snowflake?

Better than in most destinations, and this is the strongest argument for Snowflake on this particular pipeline. API responses nest, Airbyte preserves that structure rather than guessing at a flattening, and Snowflake queries semi-structured data natively rather than treating it as an opaque string.

That changes the modelling approach. Rather than flattening upstream to fit a relational destination, you can land the payload as it arrived and extract fields in views, keeping the raw structure available. When the vendor adds a field, you reprocess history rather than starting from the day you noticed, which on an API you do not control is a genuine advantage.

The discipline worth keeping is to still build the views. Landing everything as semi-structured data and telling analysts to navigate it themselves is how a useful dataset becomes one nobody uses. Extract the fields people query into a clean layer and leave the raw as the audit trail.

What happens when the API changes?

This is the question that decides whether building your own connector was the right call, and it deserves an honest answer: you own it. A maintained catalogue connector gets updated when a vendor changes their API. One you built in Connector Builder gets updated when you notice.

Two things reduce the pain. Because the connector is declarative rather than code, changes are usually a field edit in the Builder interface rather than a development task, which means someone other than its original author can fix it. And because it lives in your workspace alongside everything else, its failures surface where your other pipeline failures do rather than in a cron log nobody reads.

If the connector turns out to be broadly useful, it can be contributed back to the Airbyte catalogue, at which point maintenance stops being solely yours. Worth considering for anything pointed at a well-known public API rather than an internal service.

Frequently asked questions

Is the Public APIs connector what I want?

Only if you specifically want data from publicapis.org. For your own API or a third party's, use Connector Builder.

Why does my custom connector return only one page?

Pagination is misconfigured. Check whether the API uses page numbers, offsets or a cursor token, and whether that token comes back in the response body or a Link header.

Should I flatten the API response before loading?

Not for Snowflake. It queries semi-structured data natively, so land the payload as it arrived and extract fields in views, keeping the raw available for reprocessing.

Who maintains a connector I build myself?

You do. Because it is declarative rather than code, fixes are usually a field edit rather than a development task, and useful connectors can be contributed back to the Airbyte catalogue.

Can I do this without writing code?

Yes. Connector Builder is a user interface and the whole setup above is UI-driven. What you need is the ability to read the API's documentation, which is the harder skill.

Get your REST API data into Snowflake

Check the catalogue first, because a maintained connector beats one you own. If you do have to build, the work is reading the documentation closely enough to know the pagination scheme and whether the API will tell you what changed. Then let Snowflake hold the nested payload as it arrived and do the extraction in views.

Airbyte's connector catalog includes 600+ pre-built connectors, so it is worth looking before you build. For the same approach pointed at other destinations, see REST API to BigQuery and REST API to PostgreSQL.

Start syncing now →