Shopify to BigQuery: How to Replicate Your Data

The Shopify connector is one of the more intricate sources in the catalogue, and the reason is that Shopify does not have one API. It has three. The connector syncs through the REST Admin API, the GraphQL API, and the GraphQL BULK API, and the documentation marks which streams use which, with those labelled GraphQL going through GraphQL or BULK and the rest going through REST.

Almost everything that surprises people about this pipeline traces back to the BULK API. Understanding how it behaves is most of the work.

How does a bulk job actually run?

A bulk operation in Shopify is asynchronous. You ask for a large result set, Shopify assembles it in the background, and the connector polls until it is ready. Two properties follow from that.

The first is concurrency. Shopify permits one bulk operation at a time per store. If you see the error that a BULK job could not be created because another job is running, something else is holding the slot. That could be a second Airbyte connection against the same store, a Shopify app, or a previous job that has not finished. Running two connections from one store on overlapping schedules will produce this reliably, so stagger them.

The second is duration. Large stores produce long jobs. The connector exposes a maximum time in seconds after which a bulk job is cancelled and retried, and a checkpoint threshold controlling how many records accumulate before the job checkpoints, configurable between 15,000 and one million. If your initial sync keeps timing out, raise the job timeout first. If it fails partway and restarts from nothing, lower the checkpoint interval so progress becomes durable.

Why is one of my streams empty?

Airbyte can only sync what your app is permitted to read, and a missing scope does not fail in the obvious place. It produces a stream that is absent or empty, which reads like a connector limitation rather than a configuration gap.

Grant the full set of read scopes listed in the connector documentation when you create the app, including objects you do not think you need yet, because adding a scope later means re-authorising.

On authentication, Cloud users should use OAuth2.0, since the documentation notes that API Password authentication is being deprecated for Cloud, though the change does not affect Open Source connections. If you are syncing a store you do not own, request access to the client's store first, or OAuth fails with an access error that looks like a credential problem.

Which stream defaults cause quiet data gaps?

Fulfillment Orders excludes closed fulfillment orders by default. If your fulfilment analysis is missing completed work, that setting is why, and there is an option in the connector configuration to include them.

Discount Codes has a synchronous variant intended for discounts carrying more than roughly a hundred redeem codes. If you run large code-based promotions and your discount data looks truncated, that variant is the stream you want.

There is also a lookback control worth setting deliberately. Given a positive number of days, the connector re-fetches records from that many days before the saved state on each incremental sync. Shopify records are frequently updated after the fact, and a lookback window catches late edits that a strict cursor would miss. For order and refund data especially, a few days of lookback is cheap insurance against a reconciliation argument later.

Setting up the BigQuery destination

The BigQuery destination uses direct-load architecture, writing straight to final tables. Start by choosing a loading method.

Batched Standard Inserts is the default and the simplest, converting inserts into managed file uploads that BigQuery cleans up automatically. GCS staging gives you control over where staging files live, at the cost of creating a bucket, an HMAC key, and granting Storage Object Admin to the same service account. Note that buckets using customer-managed encryption keys are not supported, so Google-managed encryption is required.

Your service account needs the BigQuery User and BigQuery Data Editor roles. Set the dataset location to match the other datasets you will join against, because BigQuery queries can only reference datasets in the same physical location and the location cannot be changed after creation.

Output tables are partitioned daily on the extraction timestamp and clustered on that column plus the primary keys. Filtering on the partitioning column in your queries prunes partitions and reduces cost, which matters on an order table that grows indefinitely.

Frequently asked questions

Why do I get an error saying another job is running?

Shopify allows one bulk operation per store at a time. Check for a second Airbyte connection against the same store or another app holding the slot, and stagger their schedules.

A stream is empty. Is the connector broken?

Check your access scopes first. A missing read scope usually presents as an absent or empty stream rather than an explicit permission error.

My historical data looks incomplete after a connector upgrade.

Clear the affected stream and run a sync to backfill. Clearing deletes what Airbyte wrote for that stream in the destination and then rebuilds it from the source.

How do I catch orders that were edited after creation?

Set a lookback window so each incremental sync re-fetches a few days before the saved state. Strict cursor filtering misses retroactive edits entirely.

Get your Shopify data into BigQuery

Grant every read scope, tune the bulk job timeout and checkpoint interval to your store's size, set a lookback window, and choose your dataset location deliberately. For other sources landing in the same warehouse, see our guides to PostgreSQL to BigQuery and Datadog to BigQuery.

Start syncing now →