MongoDB to Databricks: How to Move Your Data

Moving MongoDB into Databricks turns flexible documents into lakehouse tables, and the decision that shapes everything is made in the source configuration rather than the destination. You choose whether Airbyte infers a schema from your collections or hands them over untouched, and that choice determines what your analysts see.

The usual first attempt is mongoexport into object storage on a cron. It gets the first copy across, re-reads everything each time, and never tells you what was deleted. This guide covers the managed path with Airbyte, using change data capture, and the prerequisites that stop this connector at setup rather than at runtime.

MongoDB to Databricks at a glance:

Why move data from MongoDB to Databricks?

Two situations account for most of these pipelines.

The first is analytical load. MongoDB is serving the application well, and then someone starts running aggregation pipelines across whole collections during business hours. Replicating into Databricks moves that work onto separate compute, so a slow analytical query stops being an incident for the people using your product.

The second is joining and machine learning. Document data next to your warehouse tables, in a form Spark and notebooks can work with, under Unity Catalog governance. If you only need a handful of current values, querying MongoDB directly is simpler and this pipeline is unnecessary machinery.

What do you need before you start?

The MongoDB source has hard prerequisites, and most failed setups trace back to one of them:

A replica set. Airbyte reads changes from MongoDB change streams, which only exist on a replica set. A standalone mongod will not work. On Atlas that means a dedicated M10 tier or above, since shared tiers can fail during connection setup.

TLS and a user with readAnyDatabase. Encryption is required by the connector and is on by default for Atlas. Since connector version 2.0.0 a single source can sync collections across multiple databases, and that access needs the readAnyDatabase privilege, without which the connection fails with an authorization error. The MongoDB source docs cover Atlas and self-hosted setups.

A consistent _id type per collection. Every document in a collection must use the same data type for _id, since mixed types are unsupported. Views, capped collections, clustered collections and empty collections are also excluded.

A secondary to point discovery at, if your cluster is busy. Schema discovery runs sample aggregations against every collection in parallel, which can put real pressure on a production primary.

On the Databricks side each stream becomes a table in Unity Catalog, and the connector needs permission to create Volumes for Avro staging. Agree the catalog and schema before the first sync so tables do not appear somewhere your governance team has to ask about.

How do you build a MongoDB to Databricks pipeline in Airbyte?

Step 1: Create a read-only MongoDB user

In Atlas, add a database user under Security and then Database Access with password authentication, give it the built-in read any database role, add the readAnyDatabase specific privilege, and restrict it to the clusters you intend to replicate. Self-hosted, create the equivalent user in mongosh and enable authorization in your configuration.

Step 2: Configure the MongoDB source and choose a schema mode

Click Sources in the left navigation, then New Source, and select MongoDB, following adding a source. Supply the cluster type, connection string, database names and credentials. Then make the schema decision covered below, and consider raising the sample size from its 10,000 document default if your collections vary a great deal.

Step 3: Configure the Databricks destination

Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply your workspace details, the catalog and schema, and authentication. Airbyte creates one table per stream and preserves the raw record alongside the typed columns, which on a schemaless source is the audit trail for what the pipeline actually received.

Step 4: Create the connection with CDC

Click Connections, then New connection, select your collections and an incremental sync mode so change streams are used and deletions propagate. Full refresh re-reads a collection end to end and is checkpointed, so a network failure resumes rather than restarts, which makes it a reasonable choice for small reference collections.

Run the first sync against one or two collections and inspect the resulting tables before extending. Schema inference is the part of this pipeline most likely to surprise you, and it is far cheaper to discover that on two collections than on forty.

Should you use schema-enforced or schemaless mode?

In schema-enforced mode, the default, Airbyte samples documents, builds a field list, and lets you deselect fields you do not need. Top-level fields become typed columns in Databricks. When a field's structure disagrees with the inferred schema, say a single object where an array was expected, the connector wraps it rather than writing null. That handling is designed for occasional inconsistencies, not for fields that change type routinely.

In schemaless mode, every document produces a record with just its identifier and a data object holding the full document. Nothing is dropped and nothing is rejected. The unpacking happens in Databricks instead, which suits Spark well, and it means a producer adding a field never breaks the pipeline.

Choose enforced mode when collections are stable enough to model and your analysts want columns they can see. Choose schemaless when the documents genuinely vary, or when the application team ships schema changes faster than you can respond to them. A lakehouse is more comfortable with the schemaless option than a relational destination would be, because handling nested structures in Spark is ordinary work rather than a workaround.

What can schema discovery miss?

Discovery samples documents rather than reading the whole collection, 10,000 by default and configurable between 1,000 and 100,000. That is a sensible default and it has the consequence you would expect: a field appearing on a small fraction of documents may not be in the sample, and therefore not in the catalogue, and therefore not in Databricks.

On a collection where two percent of documents carry an optional attribute, a default sample may miss it entirely. Raising the sample size costs discovery time and reduces the risk. Reviewing the discovered schema against what your application developers say the documents contain costs a conversation and reduces it further.

Discovery also runs sample aggregations across every collection in parallel, which is worth knowing before you point it at a busy production primary. Directing it at a secondary avoids adding that load where it hurts. And re-run discovery after any deploy that adds fields, because the catalogue does not update itself.

Frequently asked questions

Do I need a MongoDB replica set?

Yes. The connector reads from change streams, which only exist on a replica set. Atlas clusters on the M10 dedicated tier or above qualify, and shared tiers may fail at setup.

Why is a field missing from my Databricks table?

Schema discovery did not see it in the sampled documents. Raise the sample size, re-run discovery, and check whether the field is sparse enough that a default sample would plausibly miss it.

Are deletes propagated to Databricks?

Yes, when the stream uses incremental CDC. Full refresh re-reads the collection instead and does not track individual deletions.

Why did my connection fail with an authorization error?

Most likely the missing readAnyDatabase privilege, which the connector needs because a single source can sync collections across multiple databases.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. If you choose schemaless mode you will unpack the data object in Spark or SQL downstream, which is ordinary lakehouse work.

Get your MongoDB data into Databricks

Confirm the replica set and the readAnyDatabase privilege first, then make the schema mode decision deliberately rather than accepting the default. Point discovery at a secondary if your cluster is busy, start with a couple of collections, and re-run discovery whenever the application ships new fields.

Airbyte's connector catalog includes 600+ pre-built connectors, so the same MongoDB source can fan out to a relational destination too. For that pairing, see MongoDB to PostgreSQL, and for another schemaless source landing in the same lakehouse, DynamoDB to Databricks.

Start syncing now →