Dremio to BigQuery: How to Move Your Data

Moving Dremio into BigQuery gives you a queryable record of your query engine. Dremio federates across sources and accumulates datasets, spaces and reflections, and after a couple of years nobody can say which of them anybody still uses.

This guide covers the managed path with Airbyte. Two things shape the build, and the first is a hard prerequisite: this connector needs a self-hosted Dremio instance. The second is that what it carries is Dremio's own catalogue rather than the data Dremio queries.

Dremio to BigQuery at a glance:

Why move data from Dremio to BigQuery?

Two situations account for most of these pipelines.

The first is governance of a platform that has grown. Which datasets and spaces exist, who created them and which have been untouched for a year are ordinary database questions once the catalogue is in a database, and awkward ones while it lives behind an interface built for browsing.

The second is joining that catalogue to your own platform data, such as cost or usage records kept elsewhere. What this is not is a way to move the data Dremio queries, which is a different exercise entirely, and if you only need a small catalogue somewhere operational then Dremio to PostgreSQL is lighter than a warehouse.

What do you need before you start?

Four things, and the first rules some organisations out entirely:

A self-hosted Dremio instance. The connector is built on the Dremio REST API and needs an instance you can reach, so this is a deployment question rather than a configuration one. The Dremio source documentation states the requirement plainly.

An API key, from an account with the right visibility. The key carries the access of whoever issued it, so a key from a narrowly scoped account produces a narrowly scoped catalogue, and nothing downstream reveals the boundary.

Agreement about what this delivers. The REST API exposes Dremio's own metadata. Somebody expecting the contents of the datasets Dremio federates is expecting a different pipeline.

A BigQuery dataset in the right location. Location is fixed at creation and BigQuery will not join across locations, so put this where the platform data you intend to join already sits.

If your Dremio instance restricts traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Dremio to BigQuery pipeline in Airbyte?

Step 1: Confirm you are self-hosted and can reach the API

Check this before anything else, because it is a hard stop rather than an inconvenience. The connector talks to the Dremio REST API on an instance you run, so a managed deployment without that access means finding another route. Call the API once with your key to confirm both the reachability and the credentials, which separates a network problem from a permissions one before you have built anything.

Step 2: Configure the Dremio source

Click Sources in the left navigation, then New Source, and select Dremio, following adding a source. Supply your instance details and API key, then select from the available streams. Record which account issued the key, since that account's visibility is the boundary of your dataset.

Step 3: Configure the BigQuery destination

Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project identifier, dataset and service account credentials. A catalogue is small by warehouse standards, so batched standard inserts are ample and staging is unnecessary.

Step 4: Create the connection and keep the snapshots

Click Connections, then New connection, select your streams and a sync mode. Daily is generous for a catalogue that changes when somebody creates a dataset. Because the interesting questions are about change over time, accumulating rather than overwriting is usually the better choice.

Then build views, because a pile of snapshots is not something anybody wants to query directly.

Why does this need a self-hosted instance?

Because the connector is built directly on Dremio's REST API, and reaching that API requires an instance you control. The documentation states the requirement without ambiguity, which makes this one of the easier eligibility questions in the catalogue and one of the more final ones.

It also narrows who this article is for, and that is worth saying rather than implying. Organisations running Dremio themselves, often alongside a lake they also operate, can use this. Those on a managed deployment without REST access need a different approach, and discovering that at the configuration stage wastes a morning.

The API key's scope is the related detail. It carries the visibility of the account that issued it, so a key created by someone who can see part of the platform produces a catalogue describing part of the platform, and the gap is invisible from the data. Use an account whose access matches the governance question you are answering, and record which one it was beside the tables.

What should a warehouse do with a catalogue?

Answer questions about the estate rather than about the business. What exists, who made it, when it last changed and how much of it is duplicated: none of that is what people usually mean by data in a warehouse, and all of it is genuinely hard to establish from inside a platform once it has grown past a few hundred objects.

The most useful version of this accumulates rather than overwrites, because the interesting questions are comparative. When did this dataset appear, how fast is the estate growing, which objects have not changed in a year and are therefore candidates for removal. Tables arrive partitioned on the extraction timestamp, which turns repeated snapshots into dated observations without anybody designing that.

Expose two views over it: one for the current picture and one across the history, both filtering on the partitioning column so the cost stays trivial. And consider joining this to your own platform records, since a catalogue on its own describes what exists while your billing or usage data explains what any of it costs, and the combination is what turns a governance exercise into a decision anybody acts on.

Frequently asked questions

Does this work with a managed Dremio deployment?

The connector needs a self-hosted instance because it calls the Dremio REST API directly, so check your deployment before planning anything around it.

Does this move the data Dremio queries?

No. It carries Dremio's own catalogue. Replicating the underlying datasets means connecting to the systems Dremio federates over, which is a different project.

Why is part of my platform missing?

The API key carries the visibility of the account that created it. Use an account whose access matches the question you are answering, and document which one.

Should I overwrite or accumulate?

Accumulate, in most cases. The valuable questions are about change over time, and the extraction timestamp partitioning makes repeated snapshots easy to work with.

Can I do this without writing code?

The pipeline, yes, and it is a short setup. The current and historical views are modest SQL and they are what makes the catalogue answerable.

Get your Dremio data into BigQuery

Confirm you run Dremio yourself and can reach its REST API, since that is a hard stop. Be clear that this carries the catalogue rather than the data Dremio queries, and note which account issued the API key because its visibility is your dataset's boundary. Then accumulate snapshots rather than overwriting, expose a current view and a historical one, and join the result to your own platform records to make it actionable.

Airbyte's connector catalog includes 600+ pre-built connectors, so a data platform can be governed like any other system. For the same source into a column store, see Dremio to ClickHouse, and for another catalogue source into the same destination, Metabase to BigQuery.

Start syncing now →