Article

From API Docs to Data Pipelines in Minutes: How Airbyte 1.0 Unlocks the Long Tail

John Lafleur

•

Michel Tricot

•

September 24, 2024

•

7 min read

Summarize this article with:

When we launched Airbyte, our goal was simple: solve the data integration challenge for every team. With Airbyte 1.0, we’re taking a major step forward, especially in addressing one of the toughest issues: the long tail of data connectors.

As new tools and APIs emerge daily, data engineers constantly battle to keep up. Most data integration platforms ignore these less common connectors, leaving teams to build custom solutions. Airbyte 1.0 changes that with a Marketplace of connectors and an AI-powered Connector Builder that creates custom connectors instantly from API documentation links. This article dives into how this eliminates the need for in-house pipelines.

But first, what’s the problem of in-house pipelines?

The unscalable problem of in-house pipelines

When companies need to move data from various sources, they often rely on pre-built connectors for mainstream platforms like Salesforce, HubSpot, and Google Analytics. But what happens when you need to pull data from a niche platform, an internal tool, or a rapidly evolving API, that closed-source data movement solutions will most likely not support?

Building in-house ETL pipelines may seem like a quick fix, but it often snowballs into a maintenance nightmare. What starts as a “simple” script to pull data quickly turns into a full-blown engineering project as you add scheduling, monitoring, and error handling—things you never anticipated. And if you want to deliver data accurately on time, you will need to add a lot more reliability features on top of the pipelines, such as checkpointing, resumable full refreshes, avoiding stuck syncs, pagination, API rate limiting adaptation, etc. And let’s be real, ETL scripts break often, because you’re at the mercy of external APIs and data sources. Every time the schema changes, or a source adds new fields, you’re back in the trenches debugging and rewriting.

And the problem only gets worse as you need to support more sources. Studies indicate 44% of data engineers time is spent maintaining in-house pipelines, even though they tried to get as many pipelines covered as possible with existing ETL solutions. Furthermore, when the company grows, the data team will start building a self-service analytics platform for the rest of the organization. That’s when more and more custom/niche sources are added and this approach becomes non-scalable.

So what’s the solution? Airbyte 1.0 is with its new Marketplace and Connector Builder AI Assistant:

Let’s dive deeper.

From API docs to connectors in seconds with the AI Assistant

We know that building a data connector from scratch can be daunting, even with a well-documented API. But what if writing the code to manage authentication, pagination, and rate limits, along with error handling and testing, and much more, could be abstracted in a simple tool to use? So that the only time spent building that new connector is on specifying what you want the connector to do and where to fetch the data.

This is why we released the Connector Builder last year. More than 10,000 connectors were built by our community this way. What’s more, any improvement Airbyte makes on the Connector Builder will be distributed across all the connectors built with it. For instance, we increased by 4x the throughput performance last month, and this will be impacted across all those connectors.

Now imagine you only had to input the API docs link, meaning that anyone non-technical could build a new connector?

Well that’s now possible with the Connector Builder’s AI Assistant (in open beta) that Airbyte’s team built in partnership with the Fractional AI team. Here’s a quick demo:

Here’s how it works:

Input the API Docs: Simply paste the URL of the API documentation into the Connector Builder.
AI-Powered Generation: The AI parses the API documentation, automatically fills the required fields for authentication, pagination, data extraction, and more.
Instant Connector: In seconds, you have a fully functional connector, ready to sync data into your pipelines. You just need to specify which streams you want to extract data from!

This dramatically reduces the complexity of building connectors for lesser-known platforms or fast-moving APIs. What once took days could be done in minutes with the Connector Builders, and now can now be done in seconds with the AI Assistant.

We are releasing the AI Assistant in public beta, only on Airbyte Cloud for now, as it relies on our own resources Here’s a hands-on article to get started with it and share feedback to us. Our success rate for a field to fill is pretty high, but building an entire connector from scratch requires a long list of fields to fill and our success rate is then still low for the moment, but we believe that it’s worth putting it in public for the following reasons:

even when the result is wrong, most data engineers / software engineers have found it valuable as it still helps you get started and in the end build the connector in less time.
by having a lot more exposure through the community, we will be able to make it better faster through your feedback.

Our plan is to upgrade it to GA, once we’ve found that it helps non-technical people create new connectors from scratch. Right now, we’re conscious it is mostly helpful to people already building connectors.

Our end goal is it becomes your own AI-powered Integration Engineer eventually!

What’s more? The Connector Builder now also supports GraphQL APIs. Our goal will be that it also supports unstructured sources too at one point in the future.

But is that enough?

Unlocking the long tail through the Connector Marketplace

What if the community could contribute newly-built connectors to a marketplace with just one click? And what if you could use or even modify those connectors just as easily?

Now you can with the new Airbyte Marketplace, which comes with the following:

You can edit any marketplace connectors using the Connector Builder and contribute back to the marketplace in a click. This means that connector maintenance will be easily performed by the community.
Any improvement to the connector builder will affect all the marketplace connectors. For instance, we increased by 4x the throughput of our Python CDK and therefore all marketplace connectors from 2MB/s to 8MB/s at once. Our goal is that Airbyte will no longer be a bottleneck in terms of throughput performance, but the API will.
We now display usage indicators and success ones for each of the marketplace connector to give visibility over their reliability.

Imagine how many long-tail connectors have been created among the 10,000 connectors built by the community for their own needs with the Connector Builder the past year. This is how Airbyte will be able to offer 10x more connectors than any other solution out there in the next few years.

What’s next?

We’ll keep expanding the Connector Builder so it has better performances, higher reliability and supports more types of sources (including unstructured ones). We’ll keep improving the AI Assistant, so it suggests the right results for more and more types of connectors.

This means the Marketplace will also soon support more and more types of sources. We have huge goals for the marketplace. We want it to be the largest collection of connectors anywhere! To achieve this, we are thrilled to launch Hacktoberfest 2024, which will be focused on expanding our Marketplace Connector, in order to reach more than 1,000 connectors in the next few quarters. Join the competition and get a chance to win $20,000+ worth of prizes!

—-

We’re incredibly excited about what Airbyte 1.0 brings to the table. Airbyte 1.0 isn’t about adding more features for the sake of it. It’s about addressing the real pain points that slow down data workflows. We know your job is more than just syncing data—it’s about enabling insights, building models, and automating decisions. Our goal is to take care of the plumbing so you can focus on what really matters.

We can’t wait to see how the community continues to innovate with these new features. If you haven’t tried it yet, head over to our Marketplace and explore the power of AI-assisted connector building.

If you’re ready to cover all your custom data pipelines with Airbyte 1.0, get started today or join our upcoming webinar on the Marketplace and AI Assistant to learn more about the marketplace and AI Assistant.

Or you can also check the other announcements of Airbyte 1.0: