REST API to BigQuery: How to Load Your Data

Getting data from a REST API into BigQuery is the most common request in data engineering that has no single answer, because "a REST API" is not a system. It is a shape. The pipeline you build depends entirely on how the API you are dealing with handles authentication, pagination, and the question of what changed since last time.

One point of clarification first, because it costs people an afternoon. Airbyte has a connector named Public APIs, and it does not mean "any public API". It syncs the catalogue at publicapis.org, which is a directory listing of free APIs, and it has two streams and no pagination. It is a useful toy and not what you want here. For an arbitrary API you have two real options.

Option one: the HTTP Request source

Airbyte ships a generic HTTP Request source. Point it at a URL, give it credentials, and it retrieves the response. It is the fastest way to answer "is this data even useful" without building anything.

It is deliberately unsophisticated. You are not defining streams, you are not getting a modelled schema, and incremental behaviour is whatever you construct around it. Use it to explore, not to run a pipeline for two years.

Option two: Connector Builder

Connector Builder is the low-code interface for defining a real source connector. You describe the API declaratively and Airbyte generates a connector that behaves like any other in the catalogue, including in the connection setup, in scheduling, and in state management.

Four things are worth working out before you open it. Authentication comes first: bearer token, API key in a header or query parameter, basic auth, or OAuth 2.0 with a refresh flow are all supported directly, and anything genuinely unusual is worth noting now rather than discovering halfway through.

Then pagination. Page number, offset, or a cursor token returned in the response body or a Link header. Getting this wrong is the single most common reason a custom connector returns exactly one page and looks like it worked. Then streams: each endpoint returning a list becomes one, and you point Builder at the record array inside the response, since most APIs wrap results in an envelope like data or results.

Finally, incremental sync. If any endpoint accepts an updated_since filter or an equivalent, configure a datetime cursor against it. If none does, you are doing full refresh, and it is better to know that going in than to discover it when the API vendor bills you for a daily full scan. Builder's testing panel runs live requests as you configure, so you can validate pagination and auth against real responses before publishing to your workspace.

The BigQuery side

Straightforward once the source works. Supply a project, a dataset, and a service account with write access to that dataset, and use Cloud Storage staging for anything beyond small volumes.

API responses are nested, and Airbyte preserves that structure. Do the flattening in a downstream model rather than fighting the connector, and you keep the option of reprocessing from raw when the API adds a field.

Frequently asked questions

Is the Public APIs connector what I want?

Only if you specifically want data from publicapis.org. For your own API or a third party's, use Connector Builder.

Do I need to write code to build a REST API connector?

No. Connector Builder is a user interface. What you need is the ability to read the API's documentation carefully, which is the harder skill.

What if the API has no incremental filter?

Then full refresh is the honest answer. Reduce the sync frequency, and check whether a subset of endpoints can run incrementally even if the others cannot.

Can I share the connector I build?

Yes. It lives in your workspace, and it can be contributed back to the Airbyte connector catalogue.

Get your REST API data into BigQuery

The work in this pipeline is not the loading. It is reading the API's documentation closely enough to know which pagination scheme it uses and whether it will tell you what changed. Answer those two questions and Connector Builder handles the rest.

For the same approach pointed at a different destination, see REST API to PostgreSQL. And before building anything, check whether a maintained connector already covers your source: Shopify to BigQuery is what that looks like when one does.

Start syncing now →