Parquet to BigQuery: How to Load Your Files

Loading Parquet to BigQuery is one of those tasks with more than one right answer, and picking the wrong one means building a pipeline you did not need. BigQuery can already read Parquet directly, so the first question is whether a managed connector earns its place in your setup.

This guide covers when Airbyte is the better tool, how to configure the connection, and the Parquet-specific constraints that catch people out.

Parquet to BigQuery at a glance:

Should you use a pipeline or load Parquet natively?

If your Parquet files already sit in Google Cloud Storage and you want them in BigQuery, BigQuery will do that itself. It reads Parquet as a native load format and can also query files in place through external tables. For a one-off load of files already in GCS, that is simpler and cheaper than any pipeline.

A managed connector earns its place when the situation is messier. The common cases are files living in S3 or Azure Blob rather than GCS, so a native load would mean building and maintaining a transfer step first. Or a folder that keeps receiving new files and needs collecting on a schedule without re-reading everything. Or Parquet arriving alongside data from twenty other systems, where one orchestration layer beats one bespoke script plus nineteen connectors.

The short version: native loading wins for files already in GCS, and a pipeline wins for cross-cloud, recurring, or consolidated ingestion.

Which connector reads Parquet files?

There is no Parquet connector. Parquet is a format setting on a file-based source, so the connector follows the storage. For a bucket, that is the S3 source, or its GCS and Azure Blob equivalents, which share the same engine and the same Parquet options. For a single file over HTTPS or SFTP, the File source is simpler.

Create the source, enter the bucket, add a stream, set the format to Parquet, and supply a glob pattern. Patterns are relative to the bucket root, so leave the bucket name out: use **/*.parquet for every Parquet file at any depth, or events/*.parquet to scope a stream to one folder. One stream per logical table is the right granularity, because BigQuery gets one table per stream.

How do you configure the BigQuery destination?

Step 1: Create a service account

The BigQuery destination needs a service account with the BigQuery User and BigQuery Data Editor roles and its key in JSON format. Required on Airbyte Cloud, optional on open source. Most setup failures are one of those two roles missing.

Step 2: Set the dataset location permanently

You set a Project ID, a Dataset Location, and a Default Dataset ID. The location cannot be changed afterwards, so if you will join this data against existing BigQuery datasets, create everything in the same location. Getting it wrong means recreating the dataset and resyncing.

Step 3: Pick a loading method

Batched Standard Inserts turns inserts into file uploads that BigQuery loads in batches, managing and cleaning up staging files itself. It needs no extra infrastructure and suits most cases.

GCS Staging gives you control over where staging files are written, at the cost of provisioning a bucket. One constraint is absolute: the bucket must use Google-managed encryption. Customer-managed encryption keys are not supported. Check the Encryption type row on the bucket's Configuration tab before wiring it up, and confirm the service account can write to the bucket and path.

There is a mild irony worth noting: with GCS Staging, Parquet from S3 lands in GCS as staging files before loading. If your files were already in GCS, that round trip is the clearest sign you should have used a native load.

What Parquet limits should you plan around?

• Partitioned datasets are not supported. Hive-style layouts with partition values in directory names are not read as a dataset. Since BigQuery has its own partitioning model and handles Hive partitioning on native loads, this is a real gap worth weighing when choosing your approach.
• Schema inference reads one file. Columns and types come from the first matching file, the oldest written to the prefix, not from a merged view. If later files added columns, supply an explicit Input schema rather than relying on inference.
• Decimals can lose precision. The Parquet settings include a Convert Decimal Fields to Floats option, and Airbyte's docs advise against it because the conversion is lossy. Leave it off so decimals reach BigQuery intact.
• Deletes are not replicated. Incremental syncs collect new files but not removals. Deleting a file from the bucket does not delete its rows from BigQuery, so if the lake is your source of truth for deletions you need a periodic full refresh.

On naming, BigQuery conventions are stricter than most file layouts. Invalid characters become underscores, and because datasets starting with an underscore are hidden in the Explorer panel, Airbyte prepends converted namespaces with n to keep them visible. Check what was actually created after the first sync before writing queries against assumed names.

Frequently asked questions

Why not just load Parquet into BigQuery directly?

If the files are already in GCS and this is a one-off, you should. BigQuery reads Parquet natively and can query it in place through external tables. A pipeline earns its place for files in S3 or Azure, for folders that keep receiving new files on a schedule, and for consolidating Parquet with many other sources.

Does Airbyte support partitioned Parquet datasets?

No. Files matching your glob are read individually, so partition values encoded in directory names are not reconstructed as columns. If Hive partitioning matters to you, a native BigQuery load handles it and this connector does not.

Can I use a CMEK-encrypted staging bucket?

No. GCS staging buckets must use Google-managed encryption, the default for new buckets. If your policy mandates customer-managed keys, use Batched Standard Inserts, which needs no bucket of your own.

Will new files be collected automatically?

Yes, with incremental sync. Files are processed oldest to newest and Airbyte remembers what it has read, so files landing in the prefix are picked up on the next run without re-reading the bucket.

Can I change the dataset location later?

No. It is fixed at creation. Recreating the dataset in the right location and resyncing is the only fix, so decide before the first run.

Get your Parquet files into BigQuery

Decide against a native load deliberately, set the dataset location before the first sync, and check your files are not Hive-partitioned. Loading the same files into a relational database instead? See our guide to Parquet to PostgreSQL. Replicating a database into the same warehouse? See PostgreSQL to BigQuery.

Start syncing now →