Parquet File to Databricks: How to Move Your Data

Moving Parquet files into Databricks is the shortest distance between a folder in object storage and a governed table people can query. The files already carry types and columnar layout, so nothing is being converted so much as registered, catalogued and made available to everyone who should have it.

This guide covers the managed path with Airbyte. Two things shape the build: there is no standalone Parquet source, so you configure the connector for wherever the files live, and the destination needs permission to create a Volume before it can stage anything.

Parquet File to Databricks at a glance:

Why move data from Parquet File to Databricks?

Two situations account for most of these pipelines.

The first is governance. Files in a bucket are readable by whoever holds the storage credentials and invisible to everyone else, with no column-level permissions, no lineage and no catalogue. Registering them as tables in Unity Catalog turns an informal pile into something with owners, access control and a name people can find.

The second is that these files are the raw input to real work, whether transformation, modelling or training, and Spark is built for exactly that. The pairing is heavier than necessary if your files are small, uniform and only ever queried with SQL, since a lakehouse brings operational weight you would not otherwise carry; Parquet File to BigQuery is simpler for that case.

What do you need before you start?

Four things, and the third is a permission people rarely think to check:

The connector for wherever the files live. There is no Parquet source as such: you choose S3, Azure Blob Storage, Google Cloud Storage or SFTP Bulk according to where the files sit, and set the format to Parquet. The S3 source documentation covers the file-based configuration these share.

Glob patterns grouped by schema. Files that genuinely share a shape belong in one stream, and files that do not belong in another. Include the subfolder path, and remember that a double asterisk matches recursively.

Permission to create Volumes in Unity Catalog. The destination stages through Avro in a Volume, so without that permission the pipeline cannot function regardless of how correct everything else is. Ask for it explicitly, since it is not implied by the ability to create tables.

A catalogue and schema decided in advance. Each stream becomes a table in Unity Catalog, so where those tables live and who can read them is a governance decision worth taking deliberately rather than defaulting into.

If your storage account or workspace restricts traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Parquet File to Databricks pipeline in Airbyte?

Step 1: Confirm you can create Volumes

This is the prerequisite that stops pipelines dead and appears nowhere in anybody's mental model of the task. Staging happens through Avro files written into a Unity Catalog Volume, so the credentials need permission to create one. Being able to create tables does not imply it, and in an organisation where Unity Catalog permissions are administered centrally the request may take longer than the rest of the build, which is a good reason to raise it first.

Step 2: Configure the file source

Click Sources in the left navigation, then New Source, and select the connector for your storage, following adding a source. Supply the credentials for that store, then add a stream per glob group with the format set to Parquet. Because the schema is declared inside the files, you are confirming types here rather than watching a sampler guess at them.

Step 3: Configure the Databricks destination

Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply the workspace details, warehouse or cluster, catalogue and schema. Nested structures need no flattening on the way in, since Spark handles them natively, so resist the urge to pre-process files that contain arrays or structs.

Step 4: Create the connection and treat these tables as bronze

Click Connections, then New connection, select your streams and a sync mode. Incremental works from file creation or modification time, which suits a folder receiving new files. Name the tables so it is obvious they are a landing layer, because somebody will otherwise build a dashboard directly on them and be upset when the shape changes.

There is no practical ceiling on volume here, which is the main reason to choose this destination for file loads that would be uncomfortable elsewhere.

Is an embedded schema a gift or a constraint?

Both, and which one dominates depends on the destination. The gift is that types are declared rather than inferred, so a column that says it holds a timestamp holds one, and none of the sampling that CSV and JSON sources need applies here. The constraint is that the declaration is per file, so two files in the same folder can disagree and no authority exists to settle it.

Databricks is unusually tolerant of that disagreement. A landing table preserving the raw record beside typed columns absorbs variation that would break a stricter destination outright, and Spark's comfort with nested and evolving structures means a column added last quarter is an inconvenience rather than a failure. That tolerance is real and worth using.

It is not a reason to skip the glob work entirely. Files whose schemas differ genuinely and permanently, from different source systems or different export versions, still belong in separate streams and separate tables, because merging them produces a table that is a nuisance to query regardless of how gracefully it loaded. The rule of thumb is to split on deliberate differences and let the bronze layer absorb accidental ones.

What should these tables be used for?

A bronze layer, and nothing else directly. Each stream becomes a table in Unity Catalog holding what the files contained, with the raw record preserved alongside the typed columns. That preservation is the point: when somebody asks in eight months why a value looks odd, the original is still there to check rather than lost in a transformation.

The layers above are yours to build. Silver tables applying business logic, joins and quality rules, gold tables shaped for the questions people actually ask, and none of that is the pipeline's job. What the pipeline guarantees is a faithful, governed landing point that can be reprocessed whenever the logic above it changes, which is the whole argument for the medallion arrangement.

Make the layering visible through naming and permissions rather than trusting convention. A bronze table that anyone can query is a bronze table somebody will build a report on, and the first schema change in the upstream files will break that report and cost you an afternoon explaining a boundary nobody knew existed. Unity Catalog can enforce this properly, and it is worth doing at the start while the tables are new and nobody depends on them yet.

Frequently asked questions

Which connector do I use for Parquet files?

The one for wherever the files live, with the format set to Parquet. There is no standalone Parquet source, so your storage system decides the connector rather than the file format.

Why does the destination need to create Volumes?

Staging goes through Avro files written into a Unity Catalog Volume. Without permission to create one the pipeline cannot run, and that permission is separate from being able to create tables.

Should I flatten nested structures first?

No. Spark handles nesting natively, so arrays and structs can land as they are and be flattened later in a silver table if a consumer needs them flat.

Do I still need to split files by glob?

For deliberate differences, yes. Databricks tolerates incidental drift better than most destinations, but files from genuinely different sources or export versions still deserve separate streams and tables.

Can I do this without writing code?

The pipeline, yes. The silver and gold layers above the landing tables are work you do in Databricks, and they are where the value of this arrangement actually appears.

Get your Parquet File data into Databricks

Ask for permission to create Volumes before anything else, since staging depends on it and a central permissions process may be slower than the build. Group your globs by genuine schema differences and let the landing layer absorb the incidental ones. Leave nesting alone on the way in. And label these tables as a landing layer with permissions to match, because the alternative is discovering somebody's dashboard the day the upstream files change shape.

Airbyte's connector catalog includes 600+ pre-built connectors, so files in object storage can become governed tables without a bespoke loader. For the same files into a column store, see Parquet File to ClickHouse, and for another file-based source into the same destination, SFTP to Databricks.

Start syncing now →