An Easier Way to Understand Airbyte Synchronization through Events

Demystifying Airbyte Synchronization through Events - A simpler way to understand data synchronization.

ENGINEERING

May 31, 2023

2 min read

Benoit Moriceau

Summarize with AI:

An Easier Way to Understand Airbyte Synchronization through Events

Twice a year (soon to be changed to quarterly), Airbyte gives us the opportunity to spend 2 days working on a hack project. This article is about presenting one of those. Usually, those projects are POC and there should not be an expectation that what is being described will be released soon.

One of the issues that we have seen with Airbyte is that it is hard to understand what happens during a synchronization. In order to see which steps have been run, we need to go through the logs or look at the UI of our internal orchestrator to fully understand what has been run. This requires a lot of education and is hard to understand even for the Airbyte engineer. The hackday project aims to have a centralized view of what is happening within a connection. The goal is to remove the need to look at the airbyte logs in most of the use cases.

In order to do that, we added an API to add events and list events. It makes it easy to send an event either from the data plane or the control plane. Thanks to that, we can have a very granular level of error and through this experimentation we managed to have an easier way of understanding why a connection failed.

This POC could also be used for auditing, by sending an event when a connection configuration is updated.

To summarize, in 2 days we have proved that a better way to expose what events happen in a connection is possible and valuable. Ideally this could replace the existing log view and could cover most of the investigation needs. You can find the POC UI we have come up with during our hack days.

https://www.youtube.com/watch?v=yq2IVQGUECk

Integrate with 700+ apps using Airbyte

Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.

Try it free Talk to sales

About the Author

Benoit Moriceau

Integrate with 700+ apps using Airbyte

Try Airbyte for free

An Easier Way to Understand Airbyte Synchronization through Events

Integrate with 700+ apps using Airbyte

Integrate with 700+ apps using Airbyte

Related posts