n8n to BigQuery: How to Move Your Data

Moving n8n into BigQuery gives you something the n8n interface cannot: a queryable history of how your automations have actually behaved. Which workflows fail most often, whether reliability is drifting, how long runs take now compared with three months ago. The interface shows you recent executions. A warehouse lets you ask questions across all of them.

This guide covers the managed path with Airbyte. Two characteristics shape the whole design, and neither is a problem once you know: the connector exposes a single stream, executions, and it supports full refresh only. In a warehouse that combination is workable, and it turns your sync mode choice into a decision about whether you are building an archive or a mirror.

n8n to BigQuery at a glance:

Why move data from n8n to BigQuery?

Two situations account for most of these pipelines.

The first is reliability analysis. Once n8n is running dozens of workflows that other teams depend on, someone will ask which ones fail most, whether failures cluster at particular times, and whether last quarter's reliability work actually helped. Those are aggregate questions over a long window, and a warehouse answers them in SQL while an execution list does not.

The second is retention. n8n instances are commonly configured to prune execution history so the database does not grow without limit, which is sensible operationally and means the record you want for trend analysis is being deleted on a rolling basis. A warehouse is where that history can live permanently, and that possibility depends entirely on the sync mode you choose.

What do you need before you start?

The setup is genuinely simple. The expectations are the part worth getting right:

An n8n API key and your instance host. Create the key under Settings and then API in n8n. Because n8n is frequently self-hosted, the host is whatever address your instance answers on rather than a fixed vendor domain.

Realistic expectations about the stream. The connector exposes executions, the record of workflow runs. Workflow definitions, credentials, users and tags are not available. So you get the runs but not the catalogue of what was run, which shapes your modelling.

Monitoring on the connection itself. This is a Marketplace connector carrying a low sync success rate in Airbyte's own metadata. That is worth knowing rather than glossing over: build alerting on sync failures instead of assuming that silence means success.

A BigQuery service account and dataset. With the BigQuery User and BigQuery Data Editor roles. Set the dataset location to match anything you intend to join against, since it cannot be changed after creation.

If your n8n instance sits behind a firewall or in a private network, make sure it is reachable from wherever Airbyte runs, and add the Airbyte Cloud IP addresses to the allow list if inbound traffic is restricted.

How do you build an n8n to BigQuery pipeline in Airbyte?

Step 1: Check your n8n retention settings

Find out how long your instance keeps execution records before pruning them. That number tells you two things: how much history your first sync will capture, and how often this pipeline needs to run so nothing is deleted between syncs. If executions are pruned after seven days and you sync fortnightly, you are losing half of them permanently.

Step 2: Configure the n8n source

Click Sources in the left navigation, then New Source, and select n8n, following adding a source. Supply the API key and host. Airbyte tests the connection, and because this is a self-hosted tool more often than not, a failure here is usually networking rather than credentials.

Step 3: Configure the BigQuery destination

Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project, dataset and service account key. Batched standard inserts is the default loading method and is ample here, because execution records are small and this is not a high-volume pipeline by warehouse standards.

Step 4: Choose append or overwrite deliberately

Click Connections, then New connection, and choose a sync mode. This is the decision that defines what the pipeline is for. Full refresh overwrite gives you a mirror of whatever n8n currently holds, which is simple and loses everything n8n has pruned. Full refresh append accumulates, so your warehouse outlives n8n's retention and becomes the archive.

Append has a consequence to handle: because every sync re-reads all retained executions, the same execution arrives repeatedly, once per sync while it remains in n8n. Deduplicate on the execution identifier in a downstream view, keeping the most recent extraction of each. That is a few lines of SQL and it is what makes append usable.

Are you building a mirror or an archive?

Worth answering explicitly, because the two designs diverge immediately and converting between them later is not free.

A mirror is the simpler build. Overwrite on every sync, no duplicates, no deduplication view, and a table that always matches what n8n shows. It suits a dashboard about current reliability, and it is completely useless for the question about how things looked six months ago, because six months ago has been pruned from both systems.

An archive is the more valuable build and asks slightly more of you: append mode, a deduplication view, and the discipline of syncing more often than n8n prunes. In exchange your warehouse accumulates a permanent record of automation reliability that nothing else in your stack holds. If anyone has asked for trends rather than status, you want the archive, and you want it configured before the history you care about is deleted.

How do you model a single stream usefully?

With only executions available, and no workflow definitions, your records reference workflows by identifier and nothing tells you what those workflows are called. A dashboard of failure counts by workflow ID is technically correct and practically useless to the people who need to read it.

The pragmatic fix is a small mapping table in BigQuery, maintained by hand or by a lightweight job, translating workflow identifiers to names and owning teams. It feels unsatisfying to maintain manually and it takes minutes, and it turns the dataset from a technical artefact into something a team lead will actually look at.

Beyond that, the modelling is ordinary. Derive duration from start and end timestamps, flag failures, and aggregate by workflow and by day. The value of this pipeline is almost entirely in that aggregation rather than in the raw records, so build the view early and point people at it rather than at the landed table.

Frequently asked questions

Can I sync workflow definitions as well as executions?

No. The connector exposes the executions stream only. Workflow definitions, credentials, users and tags are not available, so maintain a mapping table if you need workflow names.

Can I sync incrementally from n8n?

No. The source supports full refresh only, so every sync re-reads the retained execution history. Use append mode plus a deduplication view if you want to accumulate history.

Why is my warehouse missing older executions?

Either you are using overwrite mode, which mirrors whatever n8n currently holds, or n8n pruned them between syncs. Switch to append and sync more often than your retention window.

Why do I have duplicate execution records?

Expected with append mode, because each sync re-reads everything n8n still retains. Deduplicate on the execution identifier in a downstream view, keeping the most recent extraction.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. If you choose append mode you will want a deduplication view in SQL, which is short and worth having.

Get your n8n data into BigQuery

Check your retention setting first, decide whether you are building a mirror or an archive, and sync more often than n8n prunes. Add a deduplication view if you chose append, a mapping table so workflows have names, and alerting on the connection because this connector's success rate warrants it.

Airbyte's connector catalog includes 600+ pre-built connectors, so automation reliability data can sit alongside everything else you analyse. For the same source feeding a streaming platform instead, see n8n to Kafka, and for another operational source where retention drives the design, Sentry to BigQuery.

Start syncing now →