An Easier Way to Understand Airbyte Synchronization through Events
Twice a year (soon to be changed to quarterly), Airbyte gives us the opportunity to spend 2 days working on a hack project. This article is about presenting one of those. Usually, those projects are POC and there should not be an expectation that what is being described will be released soon.
One of the issues that we have seen with Airbyte is that it is hard to understand what happens during a synchronization. In order to see which steps have been run, we need to go through the logs or look at the UI of our internal orchestrator to fully understand what has been run. This requires a lot of education and is hard to understand even for the Airbyte engineer. The hackday project aims to have a centralized view of what is happening within a connection. The goal is to remove the need to look at the airbyte logs in most of the use cases.
In order to do that, we added an API to add events and list events. It makes it easy to send an event either from the data plane or the control plane. Thanks to that, we can have a very granular level of error and through this experimentation we managed to have an easier way of understanding why a connection failed.
This POC could also be used for auditing, by sending an event when a connection configuration is updated.
To summarize, in 2 days we have proved that a better way to expose what events happen in a connection is possible and valuable. Ideally this could replace the existing log view and could cover most of the investigation needs. You can find the POC UI we have come up with during our hack days.
