MongoDB to BigQuery: How to Replicate Your Data
Replicate MongoDB into BigQuery with Airbyte. Replica set prerequisites, choosing between schema-enforced and schemaless mode, and the settings you cannot undo.
Replicating MongoDB to BigQuery is how most teams get application data in front of analysts. MongoDB serves the product; BigQuery answers the questions about it. The interesting part is not the pipeline, it is deciding how much structure to impose on documents that were designed not to have any.
That decision is a single setting on the MongoDB source, and it determines whether your analysts get clean typed columns or a JSON blob to unpack. This guide covers the prerequisites that stop setups before they start, how to choose between the two schema modes, and the BigQuery settings you cannot change later.
What does MongoDB need before you start?
Four prerequisites account for nearly every failed setup, and all of them stop the connection at configuration rather than at runtime.
- A replica set. The connector reads from MongoDB change streams, which only exist on a replica set. A standalone mongod will not work, and on Atlas this means a dedicated M10 tier or above.
- A user with readAnyDatabase. Since connector version 2.0.0 one source can sync across multiple databases, and that needs the readAnyDatabase privilege or the connection fails with an authorization error.
- A consistent _id type. Every document in a collection must use the same type for _id. Views, capped collections, clustered collections, and empty collections are excluded entirely.
- TLS. Required by the connector, and enabled by default on Atlas.
One setup behaviour is worth planning around. Because collections are schemaless, the MongoDB source samples documents to infer fields, 10,000 by default and configurable between 1,000 and 100,000. Discovery runs $sample aggregations against every collection in parallel, which is real load on a busy production cluster. Point it at a secondary if that matters.
Schema-enforced or schemaless?
This is the decision that shapes every downstream query your analysts write.
Most teams should start schema-enforced. The normalization it applies is designed for occasional inconsistencies, not for fields that change type routinely, so if a field is a string in half your documents and an object in the other half, that is a signal to go schemaless for that collection and model it properly in BigQuery instead.
What can you not change later in BigQuery?
The BigQuery destination needs a service account with the BigQuery User and BigQuery Data Editor roles and its key as JSON. You then set a Project ID, a Dataset Location, and a Default Dataset ID.
The dataset location is permanent. If your MongoDB collections will be joined against datasets that already exist in BigQuery, create everything in the same location, because fixing it later means recreating the dataset and resyncing from scratch.
For loading, Batched Standard Inserts is the simpler option and BigQuery manages its own staging files. GCS Staging gives you control over where staging files live, but the bucket must use Google-managed encryption, since customer-managed keys are not supported.
Naming is worth checking after the first sync. BigQuery conventions are stricter than MongoDB's, so invalid characters become underscores, and because datasets beginning with an underscore are hidden in the Explorer panel, Airbyte prepends converted namespaces with n to keep them visible. MongoDB database names that BigQuery dislikes will not arrive under the name you expect.
Then create the connection, select your collections, and choose incremental with CDC so deletes propagate. For more on change streams and resume tokens, see our MongoDB CDC guide. If access is restricted by IP, add the Airbyte Cloud IP addresses first.
One MongoDB limit is worth knowing before you rely on CDC: the 16MB BSON document limit can surface as a BSONObjectTooLarge error when change events carry large documents. The troubleshooting guide covers the fix.
Frequently asked questions
Do I need a MongoDB replica set?
Yes. The connector reads from change streams, which only exist on a replica set. Atlas clusters on the M10 dedicated tier or above qualify; shared tiers may fail at setup.
Which schema mode should analysts prefer?
Schema-enforced, almost always. It produces typed BigQuery columns that anyone can query without JSON functions. Reserve schemaless for collections whose documents genuinely differ from each other, and model those in dbt after landing.
Can I change the dataset location later?
No. It is fixed at creation. Recreating the dataset in the right location and resyncing is the only remedy, so decide before the first run, especially if you will join against existing datasets.
Are deletes reflected in BigQuery?
Yes, when the stream uses incremental CDC. Full refresh re-reads the collection instead and does not track individual deletions.
Why is my table name different from my collection name?
BigQuery naming rules are stricter, so invalid characters are converted to underscores and namespaces that would start with an underscore get an n prepended to stay visible in the Explorer panel. Check what was created before writing queries against assumed names.
Get your MongoDB data into BigQuery
Pick a schema mode per collection rather than globally, set the dataset location before the first sync, and use CDC so deletes follow. Sending the same collections elsewhere? See our guides to MongoDB to ClickHouse and MongoDB to PostgreSQL.
Integrate with 600+ apps using Airbyte
Move data from 600+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.