.png)
The Airbyte API is now generally available! This enables use-cases for Orchestration, Powered by Airbyte and the Airbyte Terraform SDK (more below). Let’s take a closer look at the Airbyte API and explore how it is already providing value for data engineers and developers.
What is the Airbyte API?
The Airbyte API is a developer-friendly interface that enables users to programmatically interact with Airbyte Cloud workspaces. This is in addition to the Configuration API already utilized by the OSS community (differences between these 2 APIs explored later in post).
Key benefits
1. Automation at scale
Many teams have too many sources and connections to manage manually or in the UI. These teams can leverage the API to interact with Airbyte programmatically, enabling them to manage their Airbyte Cloud workspaces at scale. This automation can help reduce errors and improve efficiency, freeing up time for data engineers and developers to focus on other high-value tasks.
2. Better integration with existing tools
The Airbyte API enables better integration with existing tools and systems within the Modern Data Stack that companies already have as part of their ELT infrastructure. For example, by integrating Airbyte with orchestration tools like Airflow, Dagster, or Prefect, this enables "Data integration as code" (example). This makes it easier to manage and monitor data pipelines and ensure data is flowing smoothly between sources and destinations.
3. Powered by Airbyte
For companies that want to provide data movement within their own products, the Airbyte API enables them to embed Airbyte's data movement capabilities directly into their product offerings. This can help companies provide a more seamless user experience for their customers, while also leveraging the power and flexibility of Airbyte's platform.
4. Terraform SDK
A Terraform SDK is being built from our Airbyte API specification. This enables teams to configure their infrastructure “as code”. Defining configurations and automating deployments. Airbyte can fit into these workflows through a future Terraform SDK (coming soon!)
- This will enable Airbyte resource settings & configurations to be version-controlled, reviewed, and tested, enabling teams to increase development speed and improve reliability, consistency, and scalability of their infrastructure and applications.
- For CI/CD pipelines, Terraform can also be leveraged by users to provision clean testing environments for integration testing and can be used to stand-up reproducible load testing environments.
Customer stories
Fillip Fleet leverages the Airbyte API to sync data from MongoDB to Snowflake within Airbyte Cloud. They also utilize the API to orchestrate their ELT workflows through Airflow in conjunction with Airbyte Cloud.
"The Airbyte API enabled us to streamline our entire ELT processing workflow without having to worry about the multiple tools we're using," says Trish Quintans, Data Engineering Lead at Fillip Fleet.
Shipyard built a new integration for their product on top of the Airbyte API. Check out their new blog post and demo video on their new Airbyte integration.
Wesley Paulson is the engineer at Shipyard who built the integration. He had this to say about it, "The developer experience using the Airbyte API was great. I thought the documentation was great and I liked how you could run sample requests from the API docs. This allowed us to create two new integrations (blueprints is what we call them) for our users to use in their data pipelines"
➡️ Get started with the Airbyte API now
Why build another API?
Historically, Airbyte released the documentation for an internal API known as the Configuration API, which was designed for communications between different Airbyte components. This API can be confusing for users, as it was originally designed for internal use by the Airbyte team. This Configuration API was also only able to be used by Airbyte OSS users.
To address these issues, Airbyte built the Airbyte API, which will provide a better developer experience and be specifically designed for customer-facing use cases. This enables Airbyte Cloud users to use an API with Airbyte for the first time.
The Airbyte API provides a curated view of the Airbyte Cloud making it easier to use and navigate. For example, developers can make real API calls to their Airbyte workspaces, through the Airbyte API docs themselves!
Conclusion
The Airbyte API provides ease of use and a great developer experience. It is quickly becoming a go-to tool for data teams thanks to its ability to manage large Cloud deployments, its integration with other tools in the Modern Data Stack, and its ability to streamline ELT workflows. With the Airbyte API now available, anyone can get started with it in combination with their Airbyte Cloud workspace (including free trials).