Firebase Realtime Database to Databricks: How to Move Your Data
Move Firebase Realtime Database into Databricks with Airbyte. Why output is two columns, full refresh only syncing, and parsing the value column in Spark.
Moving Firebase Realtime Database into Databricks is worth setting expectations about before you build it, because the shape of what arrives is unusual. Firebase Realtime Database is one large JSON tree. It has no tables and no records, so there is nothing for a connector to infer a schema from.
What the Airbyte source produces instead is two columns: a key and a value, where the value is a string representation of whatever JSON sat under that key. For a destination with weak JSON handling that is close to useless. For a lakehouse it is fine, because unpacking semi-structured data in Spark is ordinary work rather than a workaround.
Firebase Realtime Database to Databricks at a glance:
Why move data from Firebase Realtime Database to Databricks?
Two situations account for most of these pipelines.
The first is that Firebase cannot answer analytical questions about its own contents. It is built for reading and writing small pieces of a tree very quickly from client applications, not for aggregating across all of it. Anything shaped like how many, grouped by, or over time needs the data somewhere else.
The second is machine learning and joining. Application state next to your other sources, in a form Spark and notebooks can work with, under Unity Catalog governance. That combination is what makes a lakehouse the more natural destination here than a warehouse: the raw output is semi-structured, and Spark is comfortable with that.
What do you need before you start?
The list is short, and one item shapes how many connections you end up building:
A Google Cloud service account. With the Firebase Realtime Database Viewer role. Airbyte recommends a service account exclusive to it, which makes permissioning and auditing straightforward and means you can revoke access without affecting anything else.
A JSON service account key. Airbyte supports JSON keys only. Download it at creation, since that is the only time Google shows you the contents, and delete it from your machine once the source is configured. The Firebase Realtime Database source docs walk through creating both.
A decided node path, or several. The connector syncs one node path per source, defaulting to the root. Since your tree probably holds several distinct kinds of data under different nodes, plan on one source per node path you care about rather than pulling the whole root and sorting it out later.
Unity Catalog access. Each stream becomes a table in Unity Catalog and the connector needs permission to create Volumes for Avro staging. Agree the catalog and schema before the first sync rather than after tables appear somewhere unexpected.
One expectation to set with whoever asked for this: there is no incremental option. The source supports full refresh only, which shapes both your schedule and your costs.
How do you build a Firebase to Databricks pipeline in Airbyte?
Step 1: Map your tree to node paths
Look at your database and write down which nodes hold data worth analysing. A tree with users, sessions and configuration under separate nodes is three sources, three connections and three tables in Unity Catalog, which is far easier to model and govern than one enormous key-value table containing all of it. This is the design step and it takes ten minutes.
Step 2: Configure the Firebase source
Click Sources in the left navigation, then New Source, and select Firebase Realtime Database, following adding a source. Supply the database name, the contents of your service account key JSON, and the node path. Buffer size controls how many records are fetched at a time and is worth reducing if you hit memory pressure on a large node. Repeat once per node path.
Step 3: Configure the Databricks destination
Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply your workspace details, the catalog and schema, and authentication. Because the output is two columns rather than a modelled schema, the table Airbyte creates is deliberately simple, and all the structure work happens after it lands.
Step 4: Choose a schedule and a refresh strategy
Click Connections, then New connection, pick your source and destination, and choose a sync mode. Only full refresh is available, so every run re-reads the whole node. That makes frequency a real decision rather than a default: hourly syncs of a large node are hourly full reads of your production database, and Firebase charges for what you download.
Daily is enough for most analytical uses. If you choose full refresh append rather than overwrite, you also get a history of daily snapshots, which is a cheap way to get change tracking out of a source that offers none, and a lakehouse is a comfortable place to keep them.
How do you turn two columns into something usable?
The key column holds the keys of the JSON object at your node path. The value column holds a string representation of whatever was under each key, which may be a scalar or an entire nested object serialised as text. So a node holding user records gives you one row per user ID with the whole user object as a string beside it.
This is where the lakehouse earns its place. Parse the value column into a struct and project the fields you query into a modelled table, either with Spark or in SQL. Because Databricks handles semi-structured data natively, that is a normal transformation rather than a workaround, and it fits the medallion pattern most teams already use: the landed table is bronze, your parsed table is silver.
Keep the raw table. When someone adds a field to the Firebase tree, you reprocess history from what you already hold rather than starting from the day you noticed. On a source with no schema and no change tracking, that raw layer is the only record of what the pipeline actually received.
What does full refresh only mean in practice?
It means the connector has no way to ask Firebase what changed, so every sync reads the entire node path from the start. There is no cursor to nominate and no change stream to attach to.
Cost and load therefore scale with your sync frequency rather than with how much actually changed, which is the opposite of how most pipelines behave. Deletions are handled correctly almost by accident, because each sync is a complete picture, so a key removed from Firebase is simply absent from the next overwrite sync.
If you need to know when something changed rather than only its current value, append each sync with its extraction timestamp and compare successive snapshots downstream. Spark makes that comparison straightforward, which is another reason this source suits a lakehouse better than a strict relational destination.
Frequently asked questions
Why does my Databricks table only have two columns?
That is the connector's output schema. Firebase Realtime Database stores JSON rather than records, so the source emits a key column and a value column holding the stringified JSON. Parse it into a modelled table downstream.
Can I sync incrementally from Firebase Realtime Database?
No. The source supports full refresh only, so every sync re-reads the whole node path. Set your schedule accordingly.
Can one source sync several node paths?
No. One node path per source, so create one source per path. They can all point at the same Databricks destination.
How are deletions handled?
Correctly, as a side effect of full refresh. Each sync is a complete picture, so a key removed in Firebase is absent from the next overwrite sync.
Can I do this without writing code?
The Airbyte setup is entirely UI-driven. Parsing the value column into a modelled table is Spark or SQL work, and it is where the actual modelling on this pipeline happens.
Get your Firebase data into Databricks
Map your tree to node paths first, create one source per path, and set a schedule that reflects the fact that every sync is a full read. Then treat the landed table as your bronze layer, parse the value column into modelled tables, and keep the raw so history can be reprocessed when the tree gains a field.
Airbyte's connector catalog includes 700+ pre-built connectors, so application state can sit alongside your other sources in one lakehouse. For the same source feeding a warehouse, see Firebase Realtime Database to BigQuery, and for another schemaless source landing in Databricks, MongoDB to Databricks.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.