Firebase Realtime Database to BigQuery: How to Move Your Data

Moving Firebase Realtime Database into BigQuery is worth setting expectations about before you build it, because the shape of what arrives is unusual. Firebase Realtime Database is one large JSON tree. It has no tables and no records, so there is nothing for a connector to infer a schema from.

What the Airbyte source produces instead is two columns: a key and a value, where the value is a string representation of whatever JSON sat under that key. That sounds unhelpful and, for some destinations, it is. For BigQuery it works well, because BigQuery has genuinely good JSON functions and the structure you need can be pulled out in a view. This guide covers the managed path and how to model the result once it lands.

Firebase Realtime Database to BigQuery at a glance:

Why move data from Firebase Realtime Database to BigQuery?

Two situations account for most of these pipelines.

The first is that Firebase cannot answer analytical questions about its own contents. It is built for reading and writing small pieces of a tree very quickly from client applications, not for aggregating across all of it. Anything shaped like how many, grouped by, or over time needs the data somewhere else, and BigQuery is the natural somewhere else if you are already on Google Cloud.

The second is joining. Your Firebase tree holds application state, and the interesting questions combine it with billing data, marketing data or support tickets that live elsewhere. Once it is in the warehouse it stops being a silo. It is worth being honest that if you only want current values for a handful of keys, querying Firebase directly is simpler and this pipeline is unnecessary.

What do you need before you start?

The list is short, and one item shapes how many connections you end up building:

A Google Cloud service account. With the Firebase Realtime Database Viewer role, which grants read access. Airbyte recommends a service account exclusive to it, which makes permissioning and auditing straightforward and means you can revoke access without affecting anything else.

A JSON service account key. Airbyte supports JSON keys only. Download it at creation, since that is the only time Google shows you the contents, and delete it from your machine once the source is configured. The Firebase Realtime Database source docs walk through creating both.

A decided node path, or several. The connector syncs one node path per source, defaulting to the root. Since your tree probably holds several distinct kinds of data under different nodes, plan on one source per node path you care about rather than pulling the whole root and sorting it out later.

A BigQuery service account and dataset. With the BigQuery User and BigQuery Data Editor roles. Set the dataset location to match the datasets you will join against, because BigQuery can only reference datasets in the same physical location and the location cannot be changed after creation.

One expectation to set with whoever asked for this: there is no incremental option. The source supports full refresh only, which shapes both your schedule and your costs.

How do you build a Firebase to BigQuery pipeline in Airbyte?

Step 1: Map your tree to node paths

Look at your database and write down which nodes hold data worth analysing. A tree with users, sessions and configuration under separate nodes is three sources, three connections and three tables in BigQuery, which is considerably easier to model than one enormous key-value table containing all of it. This is the design step, and it takes ten minutes.

Step 2: Configure the Firebase source

Click Sources in the left navigation, then New Source, and select Firebase Realtime Database, following adding a source. Supply the database name, the contents of your service account key JSON, and the node path. Buffer size controls how many records are fetched at a time and is worth reducing if you hit memory pressure on a large node. Repeat once per node path.

Step 3: Configure the BigQuery destination

Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project, dataset and service account key, and pick a loading method. Batched standard inserts is the default and simplest. Cloud Storage staging gives you control over where staging files live, at the cost of a bucket and an HMAC key, and buckets using customer-managed encryption keys are not supported.

Step 4: Set a realistic schedule

Click Connections, then New connection, pick your source and destination, and choose a sync mode. Only full refresh is available here, so every run re-reads the whole node. That makes frequency a real decision rather than a default: hourly syncs of a large node are hourly full reads of your production database, and Firebase charges for what you download.

Daily is enough for most analytical uses. If you choose full refresh append rather than overwrite, you also get a history of daily snapshots in BigQuery, which is a cheap way to get change tracking out of a source that offers none.

What do the key and value columns actually contain?

Exactly what the names suggest, and nothing else. The key column holds the keys of the JSON object at your node path, as strings. The value column holds a string representation of whatever was under each key, which may be a scalar or an entire nested object serialised as text.

So a node holding user records gives you one row per user ID, with the whole user object as a string in the second column. Nothing is lost, and nothing is usable until you parse it. This is why the destination choice matters more here than on most pipelines: a destination with weak JSON handling leaves you with two opaque columns and no path forward.

BigQuery is a good fit precisely because its JSON functions are strong. Build a view over the landed table that parses the value column and projects the fields you actually query into real columns, then point analysts at the view. Keep the raw table, because when someone adds a field to the Firebase tree you can reprocess history rather than starting from the day you noticed.

What does full refresh only mean in practice?

It means the connector has no way to ask Firebase what changed, so every sync reads the entire node path from the start. There is no cursor to nominate and no change stream to attach to.

Two consequences follow. Cost and load scale with your sync frequency rather than with how much actually changed, which is the opposite of how most pipelines behave and is why a sensible schedule matters. And deletions are handled correctly almost by accident: because each sync is a complete picture, a key removed from Firebase is simply absent from the next overwrite sync.

If you need to know when something changed rather than only its current value, the snapshot approach mentioned above is the practical answer: append each sync with its extraction timestamp and compare successive snapshots downstream. It is not elegant, and on a source with no incremental support it is the option that exists.

Frequently asked questions

Why does my BigQuery table only have two columns?

That is the connector's output schema. Firebase Realtime Database stores JSON rather than records, so the source emits a key column and a value column holding the stringified JSON. Parse it in a BigQuery view.

Can I sync incrementally from Firebase Realtime Database?

No. The source supports full refresh only, so every sync re-reads the whole node path. Set your schedule accordingly.

Can one source sync several node paths?

No. One node path per source, so create one source per path. They can all point at the same BigQuery destination.

How are deletions handled?

Correctly, as a side effect of full refresh. Each sync is a complete picture, so a key removed in Firebase is absent from the next overwrite sync.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. You will want to write a BigQuery view in SQL to parse the value column, which is where the actual modelling happens on this pipeline.

Get your Firebase data into BigQuery

Map your tree to node paths first, create one source per path, and set a schedule that reflects the fact that every sync is a full read. Then do the real work in BigQuery: a view that parses the value column into the fields people query, with the raw table kept so history can be reprocessed when the tree gains a field.

Airbyte's connector catalog includes 600+ pre-built connectors, so the same Firebase source can feed a relational destination if that suits better. See Firebase Realtime Database to PostgreSQL for that pairing, and Firebase Realtime Database to Snowflake for another warehouse.

Start syncing now →