Dremio to ClickHouse: How to Move Your Data

Moving Dremio into ClickHouse is not what the title suggests, and being clear about that first saves a wasted afternoon. Dremio is a query engine that sits over data living elsewhere, and this connector does not extract that data. It syncs Dremio's catalog: the record of which sources, spaces, folders and datasets Dremio knows about.

So this is a governance pipeline, and a narrow one. There is a single stream and it supports full refresh only. That is genuinely useful once a Dremio instance has grown past the point where anybody can describe it, and useless if you wanted the rows behind your datasets.

Dremio to ClickHouse at a glance:

Why move data from Dremio to ClickHouse?

One situation fits this well, and one does not, and the second is why most people arrive here.

It fits lakehouse governance. A Dremio instance a few years old accumulates sources, spaces and virtual datasets that nobody has a complete picture of, and a queryable copy of that catalogue lets you ask which datasets exist, how they are organised, and what has been added since last quarter.

It does not fit moving the data Dremio queries. Dremio federates over S3, databases and other sources, and if you want those contents in ClickHouse the right route is a connector pointed at the underlying system rather than at the engine sitting above it.

What do you need before you start?

A very short list, and the first item rules the pipeline out for a large group of Dremio users:

A self-hosted Dremio instance. The connector is built on the Dremio REST API and needs direct access to it, so it requires a self-hosted deployment. If you are on Dremio's managed cloud offering, this is not the route. The Dremio source documentation states the requirement plainly.

An API key and a base URL. Those are the only two configuration fields. The base URL is wherever your Dremio instance answers, which on a self-hosted deployment is your own address rather than a vendor domain.

Realistic expectations about scope. One stream, catalogs, delivered as a full table. The documentation invites requests for further endpoints, which tells you the current surface is deliberately small rather than accidentally incomplete.

A ClickHouse instance and credentials. With permission to create tables in the target database. Volume is never a concern here, since a catalogue is a few thousand rows at most even on a large instance.

If your ClickHouse instance restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Dremio to ClickHouse pipeline in Airbyte?

Step 1: Confirm this answers your question

Write down what you expect in ClickHouse when this is finished. If the answer is a list of what exists in Dremio, this connector does that. If the answer involves rows from a dataset Dremio queries, stop here and point a connector at the underlying source instead, because no configuration turns a catalogue sync into a data extract.

Step 2: Configure the Dremio source

Click Sources in the left navigation, then New Source, and select Dremio, following adding a source. Enter the API key and base URL, then run the connection test. SSL connections are supported, and because Dremio is self-hosted here a failed test is more often networking than credentials.

Step 3: Configure the ClickHouse destination

Click Destinations, then New Destination, and select ClickHouse, following adding a destination. Supply host, port, database and credentials. The destination writes through ClickHouse's native protocol into typed columns and uses the ReplacingMergeTree engine, which matters for the history approach below.

Step 4: Choose overwrite or append deliberately

Click Connections, then New connection, select the catalogs stream and a sync mode. Only full refresh is available, so the real choice is between overwrite, which mirrors what Dremio holds now, and append, which accumulates a snapshot per sync and gives you the drift over time.

Weekly is ample either way. A lakehouse catalogue changes at the speed of people creating datasets, and there is nothing to gain from syncing hourly.

What is actually in the catalogs stream?

Dremio's own map of itself. The catalog is how Dremio organises what it can reach: sources pointing at the systems it federates over, spaces and folders that group things for humans, and datasets, which are either physical tables in a source or virtual datasets defined as views on top of them.

That is the map rather than the territory, and it is a reasonable thing to want. On a mature instance nobody can enumerate the virtual datasets from memory, and questions about how many exist, where they cluster and which spaces have grown are exactly the ones an inventory answers and a browser does not.

What it will not tell you is whether any of it is used. Query history, reflections and job records are not part of this stream, so a dataset that nobody has opened in two years looks identical to one queried hourly. Treat the output as an inventory to review rather than a list of things to delete.

How do you get history out of a full refresh?

By choosing append rather than overwrite, which is the most useful decision available on this pipeline. Overwrite gives you a table matching Dremio right now, which you could equally get by looking at Dremio. Append accumulates one snapshot per sync, and the difference between snapshots is the thing worth having.

With a weekly snapshot you can answer how many datasets existed in each month, which spaces are growing and which have been static since somebody left. None of that is visible in Dremio itself, because a catalogue describes the present and keeps no record of what it used to contain.

Two practical notes. Derive a snapshot date from the extraction timestamp so each run is identifiable, since that is what you group by. And remember that ClickHouse deduplicates through background merges, so if you are relying on distinct rows per snapshot rather than accumulating them, query with FINAL where the count matters.

Frequently asked questions

Does this move the data Dremio queries?

No. The connector syncs the catalogs stream, which describes sources, spaces, folders and datasets. For the underlying rows, point a connector at the system Dremio federates over.

Does this work with Dremio's managed cloud?

No. The connector needs a self-hosted instance so it can access the Dremio REST API directly.

Can I sync incrementally?

No. Full refresh is the only supported mode, so the meaningful choice is between overwrite for a mirror and append for accumulated snapshots.

Can I see which datasets are actually used?

Not from this stream. Query history, jobs and reflections are not included, so the catalogue tells you what exists rather than what anybody opens.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. If you choose append you will want a short piece of SQL deriving a snapshot date and grouping by it.

Get your Dremio catalog into ClickHouse

Confirm you want the catalogue rather than the data, check you are self-hosted, and choose append rather than overwrite so the pipeline accumulates something Dremio does not keep. Then treat the result as an inventory for review, because it tells you what exists and not what anyone uses.

Airbyte's connector catalog includes 600+ pre-built connectors, so the systems behind Dremio can be replicated directly where you need their contents. For another governance-shaped source, see Metabase to BigQuery, and for a lake source whose data you can move, Parquet File to ClickHouse.

Start syncing now →