Airbyte 2024 Spring Release

May 30, 2024

Summarize with AI:

Here at Airbyte, we’re constantly evolving to make data integration easier every single day. That means making it faster, more reliable, and more intuitive than ever. This year, we started quarterly releases to showcase the new features. In last February, we launched PyAirbyte at the Winter Release.

With the Spring Release today, you’ll see that we’re always looking to push the boundaries of what’s possible. With that, let’s get into what we’ve got in store for you.

A Preview of our Connector Builder AI

We’re working on something that will dramatically change the way data teams (and even other business units) will meet their custom connector needs. Today, when you start a new connector in Connector Builder, you can start from scratch, or open an existing yaml file. Soon, you will be able to start with providing a link to the docs site of the API you want to work with, and Airbyte will make the connector for you. This is the Connector Builder AI.

The low-code CDK made it so you can deploy connectors using the Connector Builder in hours instead of days. With the Connector Builder AI, you will be able to deploy working connectors in minutes, instead of hours.

Unfortunately, the Connector Builder AI isn’t yet available but we wanted to share a preview of it to you:

Let us know what you think!!

Connector Builder Updates

While the Connector Builder AI is a big deal, we’re shipping improvements to Connector Builder itself every week. Here are just a few changes that make Builder significantly more flexible, and useful with even wider range of APIs:

When working on incremental streams, you can now test the stream with a set incremental stream state. Previously, you’d have to deploy your connector, perform the first sync, and if there was a bug in your incremental stream logic, you’d only learn about that on the second sync. Running tests from a given incremental state helps verify that your connector actually syncs the incremental data you’d expect.
Record Selector now supports Record Filters and Schema Normalization — this was in the low-code framework but was not available in the Builder previously. Record Filters allow you to discard some records based on a predicate. For example, if you have a Posts stream with comments_count field, and you need to fetch Comments substream, you can set up the parent stream with a record filter on comments_count != 0, and your connector won’t perform comments requests for posts that don’t have the comments, making the connector faster.
All components now support writing custom component-level yaml manifest, unlocking flexible behaviors that are not supported by the Connector Builder UI yet. For example, we’ve added JWT Authenticator! If your API uses JWT, you can now set this up in Builder. There’s no JWT option in the Authenticator dropdown, but you can write a custom yaml block to use it today.
Managing User Inputs and config values is more flexible: you can set user input variable names independently from their IDs, which helps when you’re changing things in your connector but want to keep config input names the same to avoid breaking changes.
Connector Builder now makes the connector yaml code nicely structured: stream definitions are in their own high-level block, and authenticators, paginators and such are not duplicated anymore.
Connector Builder will automatically update the CDK version your connector uses to the most recent compatible version every time you edit a connector or run a test. Python CDK follows semantic versioning, so if you’d like to update from 0.* to 1.0, you’d want to manually change the version: key — it’s the first line of your connector manifest. Airbyte Platform will now always use the right version of the CDK when running your custom connector, too.

New Features in PyAirbyte

We created PyAirbyte because we wanted everyone with access to a Python runtime to also have access to the rich library of Airbyte connectors - enabling diverse Airbyte use cases for AI and data engineers, and also for custom app developers. This is why we’re excited to announce two new features: Airbyte Cloud orchestration and local docker-based orchestration in PyAirbyte.

PyAirbyte adds Airbyte Cloud Support

With the newly added Airbyte Cloud support, PyAirbyte users can orchestrate their Cloud-hosted pipelines as well as locally-running ones. Airbyte Cloud offers a number of reliability, scalability, and security benefits - which PyAirbyte users can now benefit from as well for workloads that they want to run in the cloud, but still be orchestrated from their own code.

Run Java connectors in PyAirbyte via Docker

We built PyAirbyte from the ground up to make sure the PyAirbyte is a standalone fantastic developer experience on any Python environment, without requiring external services or runtimes. That said, we heard from some users that they would also like to be able to run connectors that themselves are not Python-native, such as our Java-based Postgres connector. We are happy to announce that Docker-based execution is now available - thanks to a community contribution. This bumps up the number of available sources in PyAirbyte from approximately 250 to now over 325! This feature is available today in experimental status, and we would welcome feedback from users. If you do have Docker access and want to try out the full set of connectors using PyAirbyte, please feel free to give it a try and let us know what you think!

The next killer feature just might be yours!

It’s not too late to join in on our June AI and Data Engineering Hackathon. We’re inviting open source developers, AI enthusiasts, and experienced data engineers alike to help us build the future of data replication and AI. As an added bonus for our top contributors, we’ll award cash prizes for two categories - one for feature development and another for content creation. Whatever your passion is, we’d love to see you join us in this exciting event.

New Snowflake Cortex Connector

For Snowflake users who are interested in building GenAI directly within their existing Snowflake accounts, we’ve launched a new Snowflake Cortex destination. This new connector helps users create their own dedicated vector store directly within Snowflake - with no coding required! It only takes a few minutes to create a new Airbyte pipeline with Snowflake Cortex, allowing you to load any of 300+ sources into your new Snowflake-backed vectorstore.

Configuration of the Snowflake Cortex Connector

The new Snowflake Cortex destination is compatible with OpenAI and many other embedding providers, and Snowflake users will soon also have the option to use Snowflake’s own Cortex text embedding capabilities rather than calling out to an external service. Either way, setup happens in minutes and incremental processing logic is handled automatically by the Airbyte protocol.

The road towards Airbyte 1.0

We are moving fast towards releasing Airbyte 1.0. Charles, our VP of Engineering, recently shared how we think about reliability in his talk on What does it mean for an ETL tool to be reliable?. We are building Airbyte 1.0 to abide by the following points:

Airbyte is infrastructure, it always works.
Airbyte strives for an identical, complete, up-to-date copy of your data in your data warehouse in a timely manner.
Airbyte minimizes the cases where #2 isn't possible.
When #2 isn't possible, Airbyte empowers the user to achieve data reliability.

Over the last year, our engineering team focused on shipping core Airbyte reliability features. You may already be familiar with how we improve the reliability and speed of incremental syncs with Airbyte’s Checkpointing, how we handle compatibility issues across data stores with Record Change History, and how we reduce the cost of typing & deduping with Destinations V2 (previously known as normalization).

Here are some exciting features that will be part of Airbyte 1.0: