Article

How We Test Airbyte and Marketplace Connectors

Augustin Lafanechere

•

Catherine Noll

•

August 14, 2024

•

10 min read

Summarize this article with:

At Airbyte, our goal is to provide our customers with all the connectors required by their data movement use cases. Our catalog currently includes over 350 open-source connectors, and we’re not planning to stop there, we want to go for the long tail of connectors! We can’t realistically maintain this sheer number of connectors in house, this is why we originally decided to go open source.

But even if we’re not maintaining all these connectors in house, we have to provide a certain level of reliability to our end users, which brings us to the connector testing topic.

As a member of our Connector Tooling team we are always challenged to find scalable solutions for testing a theoretically unlimited number of connectors (we want to get over 1000 connectors by the end of the year!). In this article, I share a high level overview of our current testing strategies to provide our community with a glimpse on our latest connector reliabilty improvements towards releasing Airbyte 1.0.

Our challenges and strategy to test connectors

No one size fits all approaches in terms of connector testing

After having shipped 350+ connectors over the last three years, we figured that there’s no “one size fits all” approach when it comes to connector testing. Although we have strict standards for our Airbyte connectors that we maintain in house, we can’t realistically expect the same level of rigor from a brand new community contributed connector.

Airbyte uses a tiered system for connectors to help you understand what to expect from a connector. In short, there are three connector support levels: Airbyte Connectors, Marketplace Connectors, and Custom Connectors. Imposing the same level of requirements to Marketplace and Airbyte connectors is counterproductive. We want to foster community contributions to marketplace connectors. If the bar to contribute a bug fix or ship of feature to a connector is too high we will discourage contribution to it. Too many requirements impedes reliability as it discourages maintenance and incremental improvements. In other words, if a developer builds a connector that works for their use case, we want to enable them to contribute it to the community without requiring too much overhead, and simultaneously want to provide them with the tools to assess the quality of the connector as it evolves - for their own sake and the sake of any new users of the connector.

Our strategy is to grow our test requirements based on the level of usage of a connector. A popular connector will get more scrutiny than a niche one.

Concrete example: our acceptance test suite is not required to be configured in CI for marketplace connectors with low usage.

We need connector and language agnostic test suites

At Airbyte we only have two types of connectors: sources and destinations. These connectors can be theoretically written in any language as long as:

Their interface follows the Airbyte protocol
Their artifact is a Docker image along with a metadata.yaml file

This means that we should create test suites that are connector and language agnostic. Our tests suite should be able to run on any given connector written in any given language.

We need to have these agnostic, higher level, test suites to guarantee consistency in terms of behavior and coverage amongst our connectors.

To create such test suites we follow the following principles:

Prepare: test inputs are connector configuration and catalog files.
Act: tests execute airbyte commands (spec / check / discover / read or write) on a Docker image.
Assert: tests perform assertion on the command’s standard output, which contains the Airbyte messages produced by the connector following the command execution.

Three flavors of connector agnostic test suites

QA checks (code)

Connector QA checks are static assets checks not executing any connector code but validating that the connector is correctly packaged to be successfully released to production.

These tests check for instance that:

The connector has an icon.
The changelog entry is up to date.
The metadata files structure is correct.

An exhaustive list is available here.

Acceptance (code) and Validation tests (code)

Connector acceptance and validation tests make assertions about the output of the release candidate version of the connector. Validation tests are the new version of the Connector Acceptance Tests (CAT). CAT are being revamped in a more compact and better documented suite which we're naming Validation tests. They guarantee that the Airbyte protocol is followed. They test things like:

Are we emitting correct record counts?
Is the connector correctly checkpointing its state?
Does the connector correctly handle errors?
Does the produced records schema match the emitted catalog?
Is the connector spec valid?

Regression tests (code)

Connector regression tests run airbyte commands on two different versions of a connector. The control version is usually the current version of a connector, and the target version is the release candidate version.

The regression tests compare the outputs of both versions and try to spot breaking changes in terms of connector behavior. These tests are great to guarantee that a new connector is not “worse” than the previous one.

Often, changes are expected between the control and target versions of the connector. For example if the connector is being updated in response to an API change that’s adding or removing fields from the payload, we’d expect the output of the discover (and possibly read) command to differ. Regression tests surface these differences and allow us to verify that changes to the connector behavior are expected.

How we run Airbyte connector test suites

All the test suites mentioned above run in CI on pull requests. But it can also be run locally with our airbyte-ci CLI. This tool enables connector contributors to get a consistent testing environment both locally and in CI. It is powered by Dagger. More details about how Airbyte leverages Dagger Cloud here.

Here are some command examples.

Running all test suites on our <span class="text-style-code">source-faker</span> connector:


airbyte-ci connectors –name=source-faker

Running all test suites on all modified connectors on your branch:


airbyte-ci connectors –modified test

Only run QA checks on source-faker:


airbyte-ci connectors --name=source-faker test --only-step="qa_checks”

What’s next

Our source connectors are fetching data from upstream services (API, database, file storage etc.). To test our connectors we need to check that this data is correctly retrieved, formatted and transferred. It means that we must have well seeded upstream services with this data. It does not scale for Airbyte to maintain a pool of sandboxes seeded with test data for all our connectors.

Although our test suites give us a lot of confidence that connectors are safe to release, we occasionally need to deploy breaking changes, or code that handles streams that aren’t available for testing.

In these scenarios, we want to roll out the connector gradually and deliberately.

Currently, we’ll often deploy a “pre-release” version of the connector to a small subset of connections, and manually confirm that the new connector version is syncing approximately the same volume of data with no errors, before deploying the new version to all of our users. A developer will literally look at logs and metrics to verify that the connector can be rolled out to all of our users. This obviously does not scale as our connector catalog grows.

To that end, we are working toward an automated gradual rollout strategy. This will allow us to release all of our connectors in a safer, more efficient way. This strategy will involve the following steps:

Creating a “release candidate” which will initially be deployed to a small subset of our users.
A process will query for the sync status for all connections on the release candidate.
If the new version is successful on the first subset of connections we will roll out to more users.
If we don’t have enough data, we wait for more syncs to complete.
Otherwise, we will automatically rollback users to the previous version if possible. For breaking changes, we’ll pause the rollout until a developer can fix the connector.
Once we’re satisfied that the connector version is stable enough, we’ll deploy the new connector to all users.

What other great readings

This article is part of a series of articles we wrote around all the reliability features we released for Airbyte to reach the 1.0 status. Here are some other articles about those reliability features: