Article

How Airbyte Improved Connector Sync Speed up to 10x Faster

Brian Lai

•

Maxime Carbonneau-Leclerc

•

February 3, 2025

•

12 min read

Summarize with ChatGPT

During the first half of 2024, the engineering team at Airbyte was laser-focused on delivering a reliable and robust open-source ELT pipeline to sync data from point A to point B. The culmination of these efforts was Airbyte 1.0 which was officially released on September 24th 2024. Following up on our efforts to improve the correctness of how our connectors extracted data, we now moved on to the next piece of the puzzle, SPEED! This blog post shows how we improved our connector sync speeds by up to 8x with just a few lines of YAML.

At the end of 2023, we released the first iteration of the Concurrent CDK. This new framework incorporated many learnings from the years of building and maintaining API sources, which at the time, only supported processing streams synchronously on a single thread. In practice, that meant sync performance was bottlenecked by the I/O of the requests made to partner APIs. To solve this problem, the concurrent engine was designed so streams can be divided into independent sub-groups (such as all records within a date range) that extract data in parallel with each other. And the result was a significant improvement on the speed of our sync jobs.

When syncing data from 7/1 to 12/31, the range is partitioned into smaller windows that can all be processed and loaded into a destination in parallel.

‍

A challenge we often face when managing Airbyte’s extensive catalog of API source connectors is how we release updates of the CDK to all connectors. This was complicated further by the Concurrent CDK because it was designed from the ground up and connectors using the existing Synchronous CDK weren’t compatible with the new engine. Because we didn’t have a clear plan for moving existing connectors, most were unable to take advantage of the speed and performance improvements of the new engine. We also started to observe that our development experience and the velocity we could develop features was getting worse because:

Implementing new features had to be done twice, once in the legacy synchronous engine and again in the concurrent engine
Developers needed to understand two different code flows which supported different features and deviate in subtle ways

At this point, we knew that we needed to prioritize the migration of all connectors to the Concurrent CDK and the subsequent deprecation of the legacy Synchronous CDK. This blog post covers how we accomplished this and our learnings along the way.

‍

How we did it

One of the primary ways that Airbyte is able to maintain such a large number of API sources is because most are expressed using our low-code framework that allows connectors to be built within the Airbyte app.By adhering to the abstractions specified in the low-code language, we shift the responsibility of implementing features like pagination away from the connector developer and onto our connector CDK. From this package, we have full control over the Python implementation and connectors follow a single code path that we maintain. This contrasts sources that individually implement Python functions for sync functionality. This means we only had only one code path to move from the legacy synchronous engine to the concurrent engine.

‍

Managing the Complexity of Large Migrations

Although this project was scoped to just the code path used by low-code connectors, this was still a very large project. The low-code framework is a mature product with a lot of functionality implemented since its creation in 2022. To manage complexity, we had to find ways to incrementally release changes on the new concurrent flow instead of one large update which would be much more error prone. And we did so by categorizing an API source connector’s streams based on their features:

Full Refresh Stream: The API endpoint only supports fetching all records at once and cannot be checkpointed based on a heuristic like a date cursor
Incremental Stream (without parents): The API endpoint supports filtering of records based on a bookmark value like a date. Partitions of data within a specific date range can be requested independently
Substream: The API endpoint extracts a subset of data based on a parent value included in the request. We illustrate a parent-child relationship by thinking about comments in GitHub issues ; we need to know the issues in order to retrieve the comments for those issues.
Incremental Substream: The API endpoint extracts a subset of data based on a parent value and also supports filtering on the provided date range
Asynchronous Streams: The API endpoint supports triggering jobs to fetch data and polling to detect when the job is complete and records can be extracted.

Categorizing the different groups of streams allowed us to identify which benefit most from being processed concurrently. In this case, incremental streams make up 30% of streams across all connectors. And when we look at the most popular connectors managed by Airbyte, that number increases to 50%. Another reason to focus on incremental streams first was because they are able to most take advantage of concurrency by processing smaller date windows in parallel. This is in contrast to full-refresh streams which cannot be separated into time windows that run in parallel. These types of engineering-led initiatives like performance improvements emphasize the importance of engineers owning business outcomes and understanding the context of the problems they are solving for customers.

Incrementally Releasing Concurrency

One of the biggest benefits to breaking stream types into groups was that it allowed for a more organized work breakdown with a quicker path to value where we could incrementally release the changes over time. We created a lightweight adapter class `ConcurrentDeclarativeSource` which was responsible for interpreting the incoming YAML manifest and instantiating legacy streams or new streams compatible with the concurrent engine. As mentioned earlier, because incremental streams make up a majority of streams on connectors, we focused our efforts on moving just those streams to concurrency first. And by following this same pattern, we could continue to migrate additional types of streams.

‍

This also illustrates why having a standardized declarative framework for expressing connector behavior is so important. Our low-code framework makes it easy for us to inspect what features a stream has according to which components are defined in a connector’s YAML manifest. This is in contrast to our Python defined connectors where arbitrary code can be implemented in many different ways making inspection inaccurate. For example, if a stream defines a cursor component under the `stream.incremental_sync` field, we know that this is an incremental stream. Or if a stream uses an `AsyncRetriever` component, we can reliably infer that this stream triggers asynchronous jobs to fetch data.

Which classes were changed

One of the first things we prioritized when architecting the project was figuring out what parts of the legacy interface we were keeping and which ones we wanted to deprecate. Establishing this upfront gave us a clear direction on our ideal code path.

The most interesting finding was the realization that the primary unit of work being processed during a sync was now partitions of the data set (either by time window or sub-groups of data) as opposed to entire streams. And the interfaces designed to operate at the stream-level vs. partition-level were the ones being deprecated. And this was reflected in how the main classes that we deprecated were the ones that were not thread-safe for parallel partition processing.

Please check out some of our recent PRs here and here if you want more detail on the code changes to move us onto concurrent processing.

Validating and Releasing Connectors

For each categorization of streams, we would implement the CDK change, perform testing on a connector, start a progressive rollout, add monitoring when needed, and release the connector. Those steps are always done on lower risk connectors. Once we were confident enough, we would do the same steps for at least one of our critical connectors.

Implementing the Change

Because we are developing features that go out to all connectors, we put an emphasis on minimizing the steps to uptake concurrency.

Ensuring that streams automatically started using concurrency once upgraded to the latest CDK was important to making the release process as easy as possible. So once we completed the work so that low-code streams ran through the Concurrent CDK, the work to update individual connectors was minimal. It consisted of:

Update the connector to the latest CDK version to leverage the new concurrency flow
Read the API documentation to establish the right amount of concurrency
Add the `concurrency_level` to the connector’s YAML configuration
Validate in production that the concurrency level defined would optimize speed without exhausting the rate limit and slowing down the sync


concurrency_level:
  type: ConcurrencyLevel 
  default_concurrency: "{{ config.get('num_workers', 10) }}"
  max_concurrency: 50

The code to enable concurrency for our Klaviyo Source

Testing

When testing low-code connectors, we have multiple testing frameworks that operate with different levels of context:

Mock Server Tests: Python integration tests (using pytest) that mock HTTP requests/responses that validate the correctness of how we make API requests and process responses. These tests are quick, but don’t protect against changes to a partner’s API
Connector Acceptance Tests: This is our end-to-end framework that ensures the compatibility with the platform and performs basic data retrieval against our partner’s API. Because they make real requests, these tests can be slow, susceptible to API outages, and mostly cover basic test scenarios.
Live Traffic Regression Tests: Compare the outputs of a control version using the latest version of a connector and a candidate version of a connector that contains our changes. Live traffic tests are great for catching subtle bugs that are only apparent when testing large amounts of data, but can be time consuming to run.

As you can see, each of these test frameworks have their own sets of benefits and tradeoffs. But combining them altogether leaves us with a very good safety net before we make the change available to the users.Progressive RolloutsAt that point, we were pretty confident the connector should be syncing correctly in most cases. However, the variety of datasets that we were exercising was still low. Hence, we made the decision to roll the change out progressively to our users, monitoring every step of the way in case we need to address cases that were not identified previously. This makes rolling back changes in the event of a regression much quicker and thankfully this was not something we needed to do.

Measuring the Results

Being a company at the center of data movement, it is rooted in our DNA to quantify the work we do in order to make data-driven decisions. And in addition to using data to prioritize what we work on, it is of equal importance that we measure the impact of our changes. Our definition of done included the creation of a dashboard showing the different percentiles for our connectors in terms of sync duration and speed. This way, we had clear numbers for how moving to the Concurrent CDK impacted end-users and the value delivered. Here are some of our metrics on the first set of connectors we released concurrency to.

Incremental Syncs over a Duration

- Connector: source-chargebee

- PR: https://github.com/airbytehq/airbyte/pull/47978

- Most impressive chart:

Parent Dependency

- Connector: source-jira

- PR: https://github.com/airbytehq/airbyte/pull/48738

- Most impressive chart:

We hope that this blog post gives you some insight into some of the work that we’re doing over at Airbyte to move data now faster than ever! Connector development is a unique domain spanning a number of different problem spaces across data engineering, partner API integration, and open source contribution. If any of this sounded interesting or if you want to develop a new connector, feel free to join the 25,000+ members on our community Slack.

‍

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 30-day free trial