Parquet to MongoDB: How to Load Your Files

Loading Parquet to MongoDB moves data from a strongly typed columnar file format into a document store. Airbyte connects the two with a file-based source and a document destination, and the setup itself is quick. What deserves your attention first is the shape the data arrives in, because it is not what most people expect.

Every stream becomes a collection, and every record becomes a document with exactly three fields.

That third row is the one to plan around. Your Parquet columns do not become top-level document fields. They land nested inside _airbyte_data, so a query that would have been a lookup on a column becomes a lookup on a path inside an object. Knowing that before you build saves rewriting your application queries afterwards.

Why load Parquet files into MongoDB?

Parquet is a storage format, not a serving layer. Files in a bucket cannot answer a lookup by key, cannot be queried by an application, and cannot be updated. Teams reach for MongoDB when analytical output needs to become something a product can read: a set of computed segments, an enriched catalogue, or reference data an application queries at runtime.

Airbyte handles the collection and scheduling, so new files landing in the bucket are picked up and written into MongoDB without anyone maintaining a load script. If your goal is analysis rather than serving, a relational or columnar destination will suit you better, and our guides to Parquet to PostgreSQL and Parquet to BigQuery cover those.

Which connector reads your Parquet files?

There is no Parquet connector. Parquet is a format setting on a file-based source, so the connector follows the storage. For a bucket that is the S3 source, or its GCS and Azure Blob equivalents, which share the same engine and options. For a single file over HTTPS or SFTP, the File source is simpler.

Add a stream, set the format to Parquet, and supply a glob pattern relative to the bucket root: **/*.parquet for every Parquet file at any depth, or products/*.parquet to scope a stream to one folder. One stream per logical entity is right, because MongoDB gets one collection per stream.

Three source-side limits apply regardless of destination. Partitioned Parquet datasets in a Hive-style directory layout are not supported. Schema inference reads only the first matching file, the oldest written to the prefix, rather than merging across files. And incremental syncs collect new files but not deletions, so removing a file from the bucket does not remove its documents from MongoDB.

Setting up the MongoDB destination

The MongoDB destination needs a server, a database to write into, and a user that can create collections and write documents. Airbyte's documentation recommends creating a user specifically for this, which keeps permissions and auditing straightforward. An SSH tunnel through a bastion is available if the instance has no reachable address.

Namespaces are supported, so you can organise collections rather than dropping everything into one database by default.

Which sync modes can you use?

Full refresh and incremental append are both supported. Incremental deduped history is not, because the connector does not support dbt, which that mode depends on.

That has a practical consequence worth thinking through. With incremental append, records accumulate, and there is no built-in mechanism to collapse an updated record onto its earlier version. If your Parquet files are append-only event data, that is exactly right. If they are periodic snapshots of the same entities, appending will give you one document per entity per sync, and you will be deduplicating in your application or with an aggregation pipeline.

For snapshot-style files, full refresh overwrite is usually the cleaner choice, since it replaces the collection and leaves you with a current view rather than a history you did not ask for.

What happens to Parquet's types?

This is the trade-off at the centre of the pipeline. Parquet carries its own schema, including decimals with defined precision and typed timestamps, and much of that fidelity is what makes the format worth using. Writing into a JSON blob inside a document does not preserve it the way a typed relational column would.

Two practical steps follow. First, leave the Convert Decimal Fields to Floats option disabled in the Parquet format settings, since Airbyte's docs advise against it and the conversion is lossy. Second, if precision genuinely matters for the values you are moving, verify a sample of documents after the first sync rather than assuming, and consider whether a typed destination would serve you better.

Where MongoDB earns its place here is serving flexible, semi-structured records to an application. If you need exact numeric fidelity for financial reporting, that is a different job.

Frequently asked questions

Why is my data nested inside _airbyte_data?

That is how the destination writes. Each document contains _id, _airbyte_emitted_at, and _airbyte_data, with your record held as a JSON object in the last of those. Write your queries against paths inside that object, or reshape the documents after loading.

Can I deduplicate records in MongoDB?

Not through the connector. Incremental deduped history is unsupported because the destination does not support dbt. Use full refresh overwrite for snapshot data, or deduplicate downstream with an aggregation pipeline.

Does Airbyte support partitioned Parquet datasets?

No. Files matching your glob are read individually, so partition values encoded in Hive-style directory names are not reconstructed. Plan to treat the files as plain Parquet and recover partition columns another way.

Will deleting a file remove its documents?

No. The file-based source supports incremental syncs but not incremental deletes. A periodic full refresh is the practical way to bring the collection back in line if files get removed.

What permissions does the MongoDB user need?

Enough to create collections and write documents in the target database. A dedicated Airbyte user is recommended so those permissions stay scoped and auditable.

Get your Parquet files into MongoDB

Plan your queries around the document shape, pick full refresh for snapshot files and append for event data, and let Airbyte collect new files on the schedule you set. Moving documents the other direction? See our guide to MongoDB to PostgreSQL.

Start syncing now →