Parquet File to ClickHouse: How to Move Your Data

Loading Parquet files into ClickHouse is a natural pairing, because both are columnar and both are built for scanning large volumes quickly. Files sitting in object storage are invisible until somebody loads them, and ClickHouse is a cheap, fast place to make them queryable.

One thing to clear up first, because it sends people looking for a connector that does not exist. There is no standalone Parquet source. Parquet is a file format, so you choose the file-based source matching where your files live and tell it to read them as Parquet.

Parquet File to ClickHouse at a glance:

Why move Parquet files into ClickHouse?

Two situations account for most of these pipelines.

The first is making a file feed interactively queryable. Querying Parquet in place is possible and it is not fast enough to sit behind a dashboard somebody refreshes. Loading into ClickHouse gives you sub-second responses over the same data at a cost that suits a product feature.

The second is consolidation with data already in ClickHouse. Partner extracts or exports from another system next to your event data means one engine answering everything. If your querying already happens in a lakehouse, registering the files as tables is cheaper than moving them.

What do you need before you start?

The first item is a choice rather than a credential, and the rest follow from it:

The right file-based source for where your files live. S3, Azure Blob Storage, Google Cloud Storage, SFTP and others each have their own connector, and every one of them can read Parquet. Searching the catalogue for Parquet finds nothing useful; searching for your storage does.

Credentials for that storage, and a glob pattern. The pattern selects which files belong to the stream, and being specific matters, because a loose pattern that catches two differently shaped feeds produces one confused table.

Confidence that your files agree with each other. Parquet carries its own schema, which is mostly a gift and becomes a problem when files in the same stream declare different ones. Ask whoever produces them whether the schema has changed over time.

A ClickHouse instance and credentials. With permission to create tables in the target database. The destination writes through the native protocol into typed columns, which is exactly what Parquet's declared types give it.

If your ClickHouse instance restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Parquet to ClickHouse pipeline in Airbyte?

Step 1: Check the files agree before you load them

Open a recent file and an old one and compare their schemas. Parquet declares its types, so this takes minutes, and it is the check that prevents the most common failure on this pipeline: a stream whose files disagree about what a column contains. If they have drifted, split them into separate streams rather than hoping.

Step 2: Configure the file-based source

Click Sources in the left navigation, then New Source, and select the connector for your storage, following adding a source. Supply the storage credentials, set the file format to Parquet, and enter your glob pattern. The connector reads the schema from the files rather than sampling them, which is the main advantage of this format over CSV.

Step 3: Configure the ClickHouse destination

Click Destinations, then New Destination, and select ClickHouse, following adding a destination. Supply host, port, database and credentials. Because Parquet arrives with declared types, the tables ClickHouse creates need less correction than they would from a source whose types were guessed.

Step 4: Create the connection and size the first run

Click Connections, then New connection, select your stream and a sync mode. Incremental works from file creation or modification time, so new files are picked up without re-reading the ones already loaded. Parquet compresses well, so a directory that looks modest can expand considerably once loaded as rows.

Think about your table engine and sort order while you are here, because on a columnar destination those choices determine query performance far more than anything in the pipeline does.

Why is Parquet's embedded schema both a gift and a constraint?

The gift is that nothing has to be guessed. A Parquet file declares its columns and their types, so a number arrives as a number and a timestamp as a timestamp, without the sampling and inference that CSV and schemaless sources depend on. That pairs particularly well with a destination that wants typed columns.

The constraint is that the schema is a fact rather than a suggestion. If the job producing your files changed a column's type last spring, you have two populations of file that genuinely disagree, and a stream reading both has to reconcile something that cannot be reconciled.

So the practical work is in the glob pattern. Where a feed has changed shape, partition the files by that boundary and load them as separate streams, then reconcile in ClickHouse where you have SQL and can decide explicitly how the old shape maps to the new.

What happens if a file gets loaded twice?

It can happen, because incremental sync works from file creation or modification time. A partner who rewrites yesterday's file in place gives it a new timestamp, so its contents arrive again, and without deduplication those rows simply accumulate.

Where your files carry a natural key, a deduplicating sync mode handles this. ClickHouse uses the ReplacingMergeTree engine for that, and it collapses duplicates during background merges rather than on write, so a query run shortly after a sync can return rows that the same query returns cleanly later.

That is expected behaviour rather than a fault. Add the FINAL operator where exact counts matter, typically behind a view, and tell whoever queries these tables which ones are safe to read directly. On file feeds specifically it is worth asking the producer whether files are ever rewritten, because that answer determines whether you need any of this.

Frequently asked questions

Where is the Parquet source connector?

There is not one, because Parquet is a file format rather than a system. Choose the connector for wherever your files live and set the format to Parquet.

What happens if my files have different schemas?

You get trouble, because Parquet declares its types rather than inferring them. Split the differing populations into separate streams with your glob pattern and reconcile them in ClickHouse.

Why does my table contain duplicate rows?

Either a file was rewritten and re-read, or ReplacingMergeTree has not merged yet. Use a deduplicating sync mode where your data has a natural key, and query with FINAL when counts matter.

How much will the data expand when loaded?

More than the file sizes suggest, since Parquet compresses well. ClickHouse compresses too, so the result is usually reasonable, but size the first run rather than assuming.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. You may want SQL in ClickHouse to reconcile files whose schema changed, and views using FINAL where exact counts matter.

Get your Parquet data into ClickHouse

Pick the source by where your files live rather than by format, compare an old file against a new one before loading anything, ask whether files are ever rewritten in place, and put exact counts behind a view using FINAL. Then size the first run knowing Parquet compresses well.

Airbyte's connector catalog includes 600+ pre-built connectors, so file feeds can reach an interactive engine and a warehouse at once. For the same format in a warehouse, see Parquet File to Teradata, and for the same destination fed by a database, CockroachDB to ClickHouse.

Start syncing now →