Parquet to PostgreSQL: How to Load Your Files

Loading Parquet to PostgreSQL means unpacking a columnar, strongly typed file format into a row-oriented relational database. The type mapping is where most of these pipelines quietly go wrong. Parquet carries its own schema, including decimals with defined precision, and a careless conversion can round your numbers before they ever reach a table.

This guide covers loading Parquet files into a Postgres destination with Airbyte: which connector actually reads the files, how schema inference works across a folder of them, and the limits worth checking before you build anything.

Parquet to PostgreSQL at a glance:

Why load Parquet files into PostgreSQL?

Apache Parquet is a storage format, not a query engine. It is compact and fast to scan column-wise, which makes it excellent for a data lake and awkward for everything else. Analysts cannot point a BI tool at a folder of files. Applications cannot do a row-level lookup by primary key. Nothing joins cleanly against your operational tables.

Loading the files into Postgres solves all three. You get a connection string, indexes, joins against data that already lives there, and SQL that any tool in the stack understands. The trade-off is that you are now maintaining a copy, which is why the sync mode you choose in step four matters more than it first appears.

Which connector reads your Parquet files?

There is no Parquet connector. Parquet is a format setting on a file-based source, so the connector you pick depends on where the files live. For a bucket, that is the S3 source, or its GCS and Azure Blob Storage equivalents, which share the same file-based engine and the same Parquet options.

For a single file reachable over HTTPS, SFTP, or GCS rather than a whole prefix, the File source is the simpler choice. The steps below use S3, since a folder of Parquet files in a bucket is the common case.

How do you load Parquet into PostgreSQL with Airbyte?

Step 1: Grant read access to the bucket

For a private bucket, create an IAM policy allowing s3:GetObject and s3:ListBucket. Both matter: object-level permissions alone are not enough to authenticate the connection, so the policy resource list needs the bucket ARN as well as the path under it.

Attach the policy to an IAM user and generate an access key, or use an IAM role. Roles are the more secure option but they are not available on the open-source platform and must be enabled by Airbyte for Cloud accounts, so most first-time setups use a user and key pair.

Two things to note before you start moving volume. Reading from cloud storage incurs egress charges, which land on your cloud bill rather than your Airbyte one. And if your Postgres instance restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to its allow list.

Step 2: Add the S3 source and define a Parquet stream

Create the source, enter the bucket name, choose a delivery method, then add a stream. Set the format to Parquet, name the stream, and supply a glob pattern telling Airbyte which files belong to it.

Globs use wcmatch syntax and are relative to the bucket root, so leave the bucket name out of the pattern. Use ** for everything, **/*.parquet for every Parquet file at any depth, or events/*.parquet to scope a stream to one folder. Multiple patterns can be combined with a pipe. One stream per logical table is the right granularity, since Postgres will get one table per stream.

Step 3: Configure the PostgreSQL destination

Add a Postgres destination with the host, port, database, default schema, and a user that can write to it. Airbyte Cloud connects over SSL or TLS by default. From version 3.0.0 the Postgres destination uses Direct Load: records go straight into final tables with no intermediate raw tables, and each table carries your columns plus _airbyte_raw_id, _airbyte_extracted_at, and an _airbyte_meta JSONB column recording per-row typing changes.

Step 4: Pick a sync mode and run

Create the connection, select your stream, and choose between full refresh and incremental. Incremental syncs files oldest to newest and tracks what it has already seen, so new files dropped into the prefix are picked up on the next run without re-reading the whole bucket. Full refresh re-reads everything each time, which is fine for a small static dataset and expensive for anything else. Set a replication frequency, save, and trigger the first sync.

What Parquet limits should you plan around?

Four constraints account for most of the surprises here.

• Partitioned Parquet datasets are not supported. If your files sit in a Hive-style layout with partition values encoded in directory names, that structure is not read as a dataset. Plan on treating the files as plain Parquet and recovering the partition columns another way.
• Schema inference reads one file. Columns and types are inferred from the first matching file, the oldest one written to the prefix, not from a merged view of all of them. If your early files lack a column that later ones added, supply an explicit Input schema instead of relying on inference.
• Decimals can lose precision. The Parquet format settings include a Convert Decimal Fields to Floats option. Airbyte's docs advise against it, because the conversion is lossy. Leave it off and let decimals land in a Postgres numeric column, where the precision survives.
• Deletes are not replicated. The S3 source supports incremental syncs but not incremental deletes. Removing a file from the bucket does not remove its rows from Postgres. If the lake is your source of truth for deletions, you need a periodic full refresh or a reconciliation step.

One destination-side limit is worth adding. Airbyte's own documentation is direct that Postgres is not a data warehouse and recommends it for small volumes of roughly 10GB or less, or for testing. Parquet compresses hard, so a modest-looking set of files can expand well past that once it is rows in a table. Postgres also truncates identifiers at 63 bytes, which can cause column-name collisions with wide or deeply nested schemas; the troubleshooting guide covers both.

Frequently asked questions

Does Airbyte support partitioned Parquet datasets?

No. Partitioned Parquet datasets are currently unsupported. The connector reads files matching your glob pattern individually, so partition values encoded in directory names are not reconstructed as columns.

How does Airbyte infer the schema from Parquet files?

From the first file matching your path pattern, specifically the oldest one written to the prefix. Later files are not merged into a superset schema. Where files vary, define an explicit Input schema so the stream does not depend on whichever file happens to be oldest.

Will new files be picked up automatically?

Yes. With incremental sync, Airbyte processes files oldest to newest and remembers what it has already read, so files landing in the prefix are collected on the next run. Deleted files are not reflected in the destination.

Do decimal values keep their precision in Postgres?

They do, provided you leave the Convert Decimal Fields to Floats option disabled. Enabling it trades precision for float performance, which is rarely the right call for financial or measurement data.

Can I load Parquet files from somewhere other than S3?

Yes. GCS and Azure Blob Storage use the same file-based engine and expose the same Parquet settings. For a single file over HTTPS, SFTP, or GCS, use the File source instead.

Get your Parquet files into PostgreSQL

A Parquet to PostgreSQL connection takes minutes to configure once bucket access is sorted, and it keeps collecting new files on its own after that. If the volume outgrows Postgres, the same source points at a warehouse destination without rebuilding anything. Moving data out of a document store instead? See our guide to MongoDB to PostgreSQL.

Connector behaviour described here reflects the S3 source 4.15.x and Postgres destination v3.0.0, verified August 2026. Check the linked documentation for current versions.

Start syncing now →