Article

Introducing Refresh Syncs

•

July 18, 2024

•

Hi everyone! Today, I’m excited to introduce a powerful new feature coming with Airbyte 1.0: Refresh Syncs. Data pipelines often encounter various issues over time, such as missing data from specific dates or sources that don’t track deletes. The refresh sync feature addresses these challenges, ensuring your data stays accurate and up-to-date. Let’s take a closer look at how it works. If you want the video version of this post, you can find that here on our YouTube channel!

Dataset in BigQuery

We'll start by examining our dataset in BigQuery. Here, I have a dataset called public with four different tables. Currently, we’re looking at the reviews table, where I’ve run a query showing the number of records and displaying a new column called generation_id from Airbyte. At this point, there are 300 records, and the generation_id value is zero. This will be important later.

Using the Refresh Sync Feature in Airbyte

Now, let’s switch over to Airbyte. In this example, we’re transferring data from PostgreSQL to BigQuery. In the connection settings, you’ll find the “Refresh your data” option. You can also trigger refreshes on individual streams from the connection status page. Clicking this brings up two methods:

Refresh Streams and Retain Records: This method keeps the records in your destination even if the source no longer contains them.
Refresh Streams and Remove Records: This method removes records from the destination if they no longer exist in the source.

The key difference is that the first method retains all records, while the second one can result in data loss if records are deleted from the source. It’s crucial to note that the “Remove Records” option is irreversible and will result in data loss if the destination doesn’t track deleted records.

Before executing a refresh, you’ll see which streams (tables) will be affected. In this example, all four tables will be refreshed. We’ll proceed with the “Refresh and Remove Records” option.

Executing the Refresh

Once the refresh is initiated, it’s queued and processed. After completion, we’ll revisit BigQuery to observe the changes.

Back in BigQuery, rerunning the query on the reviews table shows that the generation_id has incremented to one. This indicates that the refresh was successful. During a refresh, Airbyte ensures that the source sends data and completes the sync before committing it to the destination. The generation_id helps track historical data, allowing you to see what data came before and after a refresh.

If we had chosen the “Refresh and Retain Records” option, we would see 600 records, with the generation_id incrementing accordingly to distinguish old and new records.

Wrap Up

Reliability is top of mind with the launch of Airbyte 1.0 coming up. Refresh Syncs is just one of the many other features coming with Airbyte 1.0. For more detailed information on how Refresh Syncs work, make sure to check out our documentation. You can also subscribe to get notified of when Airbyte 1.0 launches here.

‍

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 14-day free trial