Article

Introducing Direct Loading in Airbyte

•

July 16, 2025

•

Summarize with ChatGPT

A couple of years ago, Airbyte made an architectural decision for its loading process, utilizing a method called Typing and Deduping. This approach has served us well, but it also had an impact on cost and performance for our users. Let’s look at what Typing and Deduping was doing, why Direct Loading is superior, and how you can save 50-70% on data warehouse costs.

Typing and deduping and its limitations

Connectors that moved data to data warehouses operated in multiple steps.

The connector wrote a “raw” table to the destination, airbyte_internal.raw_{namespace}__{stream}.

The connector ran a SQL query, loading new records from that raw table into the true “final table”, {namespace}.{stream}, which was fully-typed.

Here’s a simplified diagram demonstrating this process.

This approach was generally very successful, but the implications were significant.

The non-typed, non-deduped raw table never stopped growing, as each successive sync was added to the previous ones.
Warehouse compute spend increased due to repeatedly running SQL queries.
Syncs were slower than they needed to be.

Essentially, sync speed wasn’t improving and destination bills were going up. You didn’t like this, and we listened.

Direct loading: The replacement for typing and deduping

Airbyte's new Direct Loading methodology skips the intermediate step of storing raw tables, instead writing typed data directly to final tables. This eliminates the problem of ever-growing raw tables and reduces compute costs by handling data type conversions before the data reaches your warehouse. The end result is the same clean data you expect, but with less storage and lower costs. Depending on the use case, projected cost savings can be between 50-70%, with up to 33% improvement in refresh speed.

How direct loading works

In direct loading, connectors do the work of type-casting, not the destination. As the name might suggest, this means Airbyte transfers final, fully-typed data to the warehouse and avoids storing persistent, non-deduped raw tables or running those transformative SQL queries in your warehouse. There are some additional nuances around append, deduplication, and overwrite syncs, which the documentation explains in more detail, if you’re interested. At a high level, though, that’s all there is to it!

How we handle schema evolution

Few things in life are guaranteed like schema evolution and taxes. In the old typing and deduping world, schema changes triggered soft resets, something else that was slow and expensive, even if reliable.

Direct Loading uses alter table queries (or something equivalent as needed). There will be cases where a column type changes and historical records contain a value that can’t be cast to the new type. This will cause your syncs to fail.

Direct loading presents you with some options, here, too.

Easy, but expensive: run a full refresh with Airbyte.
Harder, but cheaper: Manually update historical records so they’re valid for that column’s new type.

If you’re not ready to move off raw tables

Although Direct Loading is a significant improvement, we understand that some people who interact with raw tables aren't ready to abandon them. No need to worry!

You can modify your connector’s configuration in Airbyte, enabling the “Legacy raw tables” option. When this is on, the connector only writes raw tables, leaving you free to keep working with them.

Use Direct Loading today

Direct Loading is now available for Airbyte’s BigQuery connector. Upgrade to the latest version of the BigQuery Connector to begin enjoying improvements in speed and reductions in warehouse bills.

ClickHouse and Snowflake are coming soon, with more destinations being added continuously.

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 30-day free trial