Company Updates

Airbyte Schema Propagation: Keeping your replicated catalog up to date

Malik Diarra
June 7, 2023
3 min read
Limitless data movement with free Alpha and Beta connectors

How does schema propagation work?

The Airbyte platform relies on the existing Airbyte protocol primitives to implement schema propagation: the same DiscoverSchema operation that is being run when a user sets up a new connection is also being run automatically before sync. The platform then compares the newly fetched schema with the one that is currently stored from replication.

Here again, relying on the existing Airbyte protocol allows us to make that functionality available to all connectors: the one provided by Airbyte, the one you created yourself with our CDK or the connection builder.

What’s next?

Our goal with the schema propagation feature is to reduce the operational load of maintaining pipelines. New ETL replications should be easy to set up and should require minimal involvement from the user to keep running: being resilient to infrastructure issue or source changes. This is an area of focus for us. If you have an idea to reduce our operational load while using Airbyte, please reach out to us on Slack.

The data movement infrastructure for the modern data teams.
Try a 14-day free trial