Firebase Realtime Database to Elasticsearch: How to Move Your Data
Index Firebase Realtime Database into Elasticsearch with Airbyte. Plan availability, choosing the node path, declaring a mapping, and handling deletions.
Indexing Firebase Realtime Database into Elasticsearch gives an application search that Firebase itself cannot provide. Realtime Database is built for fetching data by path and syncing it to clients, and it offers nothing resembling full-text search, relevance ranking or typo tolerance.
Two things decide whether this suits you. The Elasticsearch destination runs on Airbyte Core and PyAirbyte only, and Realtime Database is a single JSON tree rather than a set of tables, so what you index depends entirely on which part of that tree you point the connector at.
Firebase Realtime Database to Elasticsearch at a glance:
Why move data from Firebase to Elasticsearch?
Two situations account for most of these pipelines, and both are about a capability Firebase lacks.
The first is search inside your product. Users typing fragments, misspelling names or searching across several fields at once is exactly what a search engine does and exactly what Realtime Database cannot do, since its queries work by path and simple ordering rather than by relevance.
The second is operational lookup, where support staff need to find a record by a fragment of an address or a partial name. If you want counts and trends over your application data instead, a warehouse or lakehouse is the better destination.
What do you need before you start?
The first item can rule the pipeline out, and the second decides what you are actually indexing:
A self-managed Airbyte deployment. The Elasticsearch destination is available on Airbyte Core and PyAirbyte, and not on the paid Cloud tiers. No configuration works around this.
A decision about which node to index. Realtime Database is one deeply nested JSON tree, and the node path you give the connector determines what each document contains. Choosing the wrong level produces either enormous documents or fragments that mean nothing on their own.
A Google service account with read access. Along with your database name. Create one for the pipeline rather than reusing something your application uses, so its access is separately revocable.
An index mapping you have designed. This matters more here than on most sources, because a schemaless source feeding dynamic mapping means two systems guessing in sequence. Decide which fields are analysed text and which are keyword fields before indexing anything.
Finally, know what your application writes into that node, because nothing in the pipeline will tell you. The developers who built it are the documentation.
How do you build a Firebase to Elasticsearch pipeline in Airbyte?
Step 1: Choose the node and design the mapping
Pick the node whose children are the things users will search for, one document per child. Then decide the mapping: names, descriptions and free text analysed for search, while identifiers, status values and categories are keyword fields for exact filtering.
Step 2: Configure the Firebase source
Click Sources in the left navigation, then New Source, and select Firebase Realtime Database, following adding a source. Supply the database name, your service account credentials and the node path you decided on.
Step 3: Configure the Elasticsearch destination
Click Destinations, then New Destination, and select Elasticsearch, following adding a destination. Supply your endpoint and authentication, and create the index with your mapping first, so the connector writes into a schema you designed rather than one dynamic mapping invented from whatever the first record happened to contain.
Step 4: Create the connection and plan for removals
Click Connections, then New connection, select your stream and a sync mode. There is no change data capture here, so a record deleted in Firebase stays in your index until something removes it, which on a search index is worse than staleness.
Set the frequency against how quickly users expect new records to become findable, which for a product search feature is usually minutes rather than hours.
Why is the mapping especially important here?
Because you have two layers of guessing stacked on top of each other. Realtime Database has no schema, so nothing declares what a field contains, and dynamic mapping in Elasticsearch then infers a type from whichever document arrives first. Neither layer knows what the field is for.
That produces a specific hazard on a schemaless source: a field that is a number in most records and occasionally a string, or absent entirely, gets mapped from the first example and then rejects or mishandles the rest. A declared mapping removes the guessing and fails predictably instead.
The analysed and keyword distinction still applies as usual. A product name or description benefits from analysis so partial searches match; a status, category or identifier should be a keyword field so filtering is exact. Getting these the wrong way round means filters returning nothing while searches match everything.
What happens to deleted records?
They stay in the index. The connector has no change data capture, so a record your application removed from Firebase remains searchable indefinitely, and a user clicking that result lands on something that no longer exists.
On a search index that is worse than stale data, because it is actively misleading rather than merely out of date. The cleanest fix is in your application: write a deleted flag into the record rather than removing the node, and filter on that flag when querying the index.
Where you cannot change the application, a periodic full refresh rebuilds the index and therefore reflects removals. That works, it costs a complete re-read of the node each time, and on a large tree it is the sort of thing to schedule overnight rather than hourly.
Frequently asked questions
Why can I not find the Elasticsearch destination in Airbyte Cloud?
It is available on Airbyte Core and PyAirbyte, and not on the Standard, Plus, Pro or Enterprise Flex plans. This pipeline needs a self-managed deployment.
How do I decide what a document should be?
By the node path you give the connector. Pick the node whose children are the things users search for, so each child becomes one document.
Do deleted records disappear from the index?
No. There is no change data capture, so write a deleted flag in your application and filter on it, or schedule a periodic full refresh.
Can I rely on dynamic mapping?
Not on a schemaless source. A field whose type varies between records gets mapped from the first example and then mishandles the rest, so declare the mapping yourself.
Can I do this without writing code?
The Airbyte setup is UI-driven. You will want to define the index mapping in Elasticsearch, which is short JSON, and possibly a change in your application to flag deletions.
Get your Firebase data into Elasticsearch
Check your Airbyte plan first, then choose the node path deliberately since it defines your documents. Declare the index mapping rather than letting two schemaless layers guess in sequence, and solve deletions with a flag in your application, because a search result pointing at nothing is worse than a stale one.
Airbyte's connector catalog includes 600+ pre-built connectors, so application data can serve search and analytics at once. For the analytical version, see Firebase Realtime Database to BigQuery, and for another source feeding a search index, MySQL to Elasticsearch.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.