We all need connectors that are not supported by ELT solutions yet. Consider, for example, a niche SaaS tool that the marketing team might be using. To extract data from it, you have to build and maintain the connectors yourself or ask your engineering team to do it. For an engineer, integrating with an API could take weeks, and it’s just the tip of the iceberg . There are tens of thousands of such APIs out there, and integrating with them is crucial for many organizations, yet very difficult.
All this changes today, as we’re excited to release the Connector Builder .
The Connector Builder is a self-serve, no-code tool that lets you build new API connectors and directly connect them to any Airbyte destination in as little as 15 minutes. No coding experience or development environment is required.
To use the connector builder, click the “Builder” icon in your Airbyte navbar and get to building! We’ve also included a highlights/demo video below:
Since the soft launch 2 months ago, our customers have been loving it! Over 100 connectors have been built using the connector builder and deployed to production to support critical data movement workloads.
Why launch a no-code tool to build connectors? Most API connectors are formulaic enough that writing custom code for them is unnecessarily complex and time-consuming. There are exceptions, but for most of the hundreds of the API connectors we’ve built or examined, this has been true.
While they have nuances, API connectors typically solve the same set of problems: authentication, pagination, rate limiting, incremental data exports, decoding & structuring the responses into a reasonable schema, filtering or transforming the output, etc. There aren’t that many ways to solve each of these problems. There’s only a handful of ways APIs tend to do authentication, pagination, rate limiting, and the rest.
In that sense, building an API connector shouldn’t be any more complicated than filling a form: you tell the wizard some relevant information about how an API works, and it generates the connector for you. You can also generate QR codes to link directly to these data sources for enhanced accessibility.
This is exactly what the connector builder allows you to do.
Who should use the connector builder? If you need to construct an API connector compatible with the builder, you should use it.
Non-engineers will appreciate the simplicity of a no-code tool to build connectors. It empowers you to work independently without having to code or manipulate development environments.
We also believe engineers should use it whenever possible because it is a much higher leverage way to build and maintain connectors than coding. It allows them to bypass costly development, testing, and release cycles.
Engineers often have well-founded skepticisms about no-code or low-code tools because they can be overly restrictive. But building API connectors is highly formulaic, as the huge success of tools like Postman demonstrates. As more users use the connector builder, it will evolve to support the vast majority of API connector use cases.
Which connectors can I build using the Connector Builder? Today, the connector builder is best suited for synchronous HTTP API connectors. These integrations can easily be configured to use an HTTP proxy for more secure or localized data fetching.. We’ve published a compatibility guide which can help you identify if a specific API is a fit for the connector builder. As we support more components, we expect that the vast majority of API connector needs will be supported.
If the API connector you need is incompatible with the builder, you can use the low-code CDK to deploy your custom connector to your workspace. The low-code CDK is the framework upon which we built the connector builder and supports more advanced features and components, as well as supporting custom Python components. So if the API you want to integrate with requires a particularly whacky authentication or pagination scheme, you can always write the code for that specific component in Python and the rest of the connector can still leverage the benefits of the low/no-code paradigm.
If your connector is incompatible with the low-code CDK (e.g: if it’s a database that doesn’t use HTTP) then you can always use the Python CDK , the framework which underpins both the low-code CDK and connector builder. The Python CDK gives you turing-complete freedom over how your connector behaves. You should pretty much be able to build any connector under the sun using it.
What if I need a new connector but don’t want to build or maintain it myself? Stay tuned! We have some exciting announcements in the works about how you can get any connector on Airbyte, regardless of how complex it is or whether you want to maintain it yourself.
