Article

Airbyte’s journey until 1.0

John Lafleur

•

Michel Tricot

•

September 16, 2024

•

8 min

Summarize with ChatGPT

As I write this, we’re on the brink of something monumental: the release of Airbyte 1.0 on the 24th of September 2024. It's hard to believe how far we’ve come in just four years. When Michel and I first started Airbyte in 2020, we had a simple idea: make data integration accessible to everyone. But it’s been anything but simple. It’s been a ride filled with late nights, learning from our community, and building something that we believe will change how data moves.

Let’s take a moment to look back at the big moments that got us here - one year at a time.

2020: Starting from Scratch

2020 was the year it all began. Michel and I had both spent years in the data space, and we kept seeing the same problem — data teams struggling with in-house pipelines that are brittle, hard to maintain, gives inaccurate data inconsistently, just expensive. Michel and I had those scars too. We knew there had to be a better way. So, we took a leap and started working on Airbyte in July 2020.

What went down in 2020:

Founding Airbyte: We set out to build an open-source data movement platform. Open-source was a no-brainer for us as all the data teams that we discussed with and that were using some closed-source solution still had to build in-house pipelines on the side. We wanted to build a product with the community, not just for the community.
Launching on GitHub (October 2020): Before we released our first version v0.1 live, we called Airbyte “conduit” as you can see in our first readme! Yep, much prefer Airbyte as a name! v0.1 was bare-bones, with just 3 source connectors and 2 destinations, but the reception was immediate. Within weeks, we had developers reaching out, contributing, and asking for more. We understood that the problem we were addressing was felt by most data teams.
Seed Funding: We raised $5.2 million, which was just enough to help us start building a team around our vision. But back then, it was still just an idea taking shape.

2021: Building with the Community

2021 was when things started to get real. We knew we needed to move fast, and the community was right there with us. This was the year we really felt like we were onto something bigger.

Key moments from 2021:

Expanding the Connector Catalog: We went from 10 connectors to over 50, thanks in large part to our incredible community. We focused on the most requested data sources like MySQL, PostgreSQL, and Salesforce, and we kept pushing to add more.
Custom Connector Development Kit: We released a first toolkit that made it easy for developers to build their own connectors. This opened the door for Airbyte to support virtually any data source, and it’s still one of our most impactful moves.
Series A Funding: A $26 million raise gave us the runway to double down on development, community support, and building out Airbyte Cloud.

2022: Growing Pains and Gains

By 2022, we were starting to see serious adoption. We were no longer the new kids on the block; people were looking at Airbyte as a major player in the data integration world. But with that came a whole new set of challenges.

Notable events in 2022:

Series B Funding: Another big milestone—a $150 million raise. This wasn’t just about money; it was about scaling up for the long haul and being able to deliver on our promises to our users, as we started to understand that the technological challenge in front of us was not easy. We wanted to give ourselves time to build the platform that will solve all data integration problems.
Airbyte Cloud (April 2022): We launched Airbyte Cloud, a managed service to take the infrastructure headaches out of the equation for users. It wasn’ t far from perfect, but it was a big step in the right direction. We learned a ton about how different Airbyte use cases and expectations of Cloud users were compared to open-source users. We learned how more reliable and better performing our connectors needed to be.
Connector Maintenance Program: We had been focused mostly on expanding our catalog of connectors until then, but we learned the hard truth: our connectors were not up to expectations. So we started focusing more and more on bringing reliability and performance to our connectors. This would be key to keeping the trust of our users, but this was also a long journey, and we are thankful to our community, because their usage and feedback helped understand our gaps.
Transformation Layer: We introduced an in-flight data transformation layer, letting users do more with their data pipelines using SQL, dbt, or Python scripts. This was a game-changer for a lot of our users.
Community Contributions Surge: About 70% of our new connectors in 2022 were built by the community. That’s something I’m personally incredibly proud of. It showed us we were on the right track by putting community first.

2023: Hardening the Platform

If 2022 was about growth, 2023 was about maturity. We needed to stabilize the platform and make sure we were ready to serve not just developers and startups, but also mid-market and large enterprises.

2023 in a nutshell:

Connector Certification Program: We launched this to ensure that all connectors meet the highest standards of reliability. It became a cornerstone for larger companies looking for guarantees around quality. To be clear, this requires constant focus. Building reliable connectors at scale is very hard!
Building a reliable and scalable Cloud platform: We put a significant focus on making Airbyte Cloud more reliable and scalable. This meant improving infrastructure, reducing downtime, and ensuring it could handle the needs of both small startups and large enterprises. This essentially made us experts at self-hosting Airbyte, and enabled us to start working on a self-managed enterprise product.
Expanding beyond UI with an official API and Terraform Provider: We took a hard look at our interface and made our UI more intuitive and user-friendly. But also, we started addressing the programmatic management of pipelines through an official API and Terraform Provider. Today, this Terraform Provider has been downloaded more than 500k times! This was driven by countless feedback sessions with our users—thank you if you were one of those voices!
No-Code/Low-Code Connector Builder: We introduced a no-code/low-code connector builder, empowering non-technical users to create custom connectors without writing code. This was the biggest unlock in terms of addressing custom connectors for our community. More than 10k custom connectors have been built this way since then! But our main reason for building the Connector Builder was actually to nail the maintenance of the long tail of connectors at scale. The way to do it is to write as little custom code as possible!
Airbyte Self-Managed Enterprise (alpha): We started selling premium SLA-backed support back in June 2023. Soon after, Single Sign-On (SSO), role-based access control (RBAC), data masking, and advanced monitoring were rolled out. We learned a lot this year about what it means to support companies with big, complex data needs. At the end of the year, we had a first very early version of Airbyte Self-Managed Enterprise.

2024: Here Comes Airbyte 1.0

And here we are in 2024. It’s been a long road to get to this point, but I can honestly say that Airbyte 1.0 is by far the most exciting launch of Airbyte’s history! This is a huge milestone for us, but it’s also really just the beginning.

Highlights of 2024 until 2024:

Enhancing reliability and observability: We knew at the beginning we were getting close to our own 1.0 expectations, but if there’s one thing that is the most important to us is the community’s trust. So we doubled down on all the reliability features surrounding our pipelines: some text
- Handling large records
- Checkpointing to ensure uninterrupted data syncs
- Load balancing Airbyte workloads across multiple Kubernetes clusters
- Resumable full refresh, as a way to build resilient systems for syncing data
- Refreshes to reimport historical data with zero downtime
- Notifications and Webhooks to enable set & forget
- No stuck syncs
- Supporting very large CDC syncs
- Automatic detection of dropped records
PyAirbyte: We introduced PyAirbyte, an open-source Python SDK that allows developers to build and test connectors programmatically. It’s a major step in making connector development faster and more accessible, especially for data engineers and Python enthusiasts. It was also a way for us to start addressing the AI use case!
Supporting AI use cases with unstructured sources and vector databases: As AI applications surged, we expanded Airbyte's capabilities to handle unstructured data sources and seamlessly integrate with all the major vector databases, including Pinecone and Weaviate. This enables you to simplify your AI workflows, and support RAG architectures. In the end, it allows you to load, transform, and store data in a single operation.

All this is getting us to Airbyte 1.0, along with Airbyte Self-Managed Enterprise GA! Both will be launched on 09/24, stay tuned here. Some spoilers, but this launch includes quite a few significant industry innovations such as an AI building connectors(?!), the first Connector Marketplace… can’t wait!

Looking Ahead

Airbyte 1.0 is a significant marker for us, the whole Airbyte team can’t wait to share what we’ve got in store for you. But even after 1.0, there’s still so much more we want to do. This journey has always been about building with our community, learning from our users, and pushing the envelope of what’s possible in data movement. We’ve come a long way, but I’m excited to see where we go next.

Thank you to everyone who has been a part of this journey with us. Here’s to the next chapter!

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 14-day free trial