If you already utilize Terraform to manage your Modern Data Stack, Airbyte can now be included through our new Terraform Provider. This is a Terraform provider based on our new Airbyte API ! Save time on managing Airbyte and collaborate on Airbyte configuration changes with your teammates. Engineering teams who employ an “Infrastructure-as-Code” ethos will love the experience.
Join our team for a live demo of the Airbyte Terraform provider!
What is “Infrastructure-as-Code”? Where does Terraform fit in? Infrastructure as Code (IaC) makes it easier to create your server infrastructure. Instead of setting up everything manually, you can define your infrastructure through configuration files.
You specify the servers, storage, networks, security settings, and other necessary resources. IaC tools (such as Terraform) then interpret these configuration files and automatically provision the infrastructure according to the defined specifications. IaC also provides benefits such as version control, as the infrastructure code can be stored in a code repository and tracked over time.
An example use-case is for setting up CI/CD pipelines. Terraform can be leveraged by users to provision clean testing environments for integration testing and can be used to stand up reproducible load-testing environments.
Terraform is a leading tool in the IaC space, and can be thought of as consisting of two main parts: Terraform Core and Terraform Plugins. Core is responsible for the lifecycle management of infrastructure. It considers the current state and evaluates it against your desired configuration. It then proposes a plan to add or remove infrastructure components as needed. Next, it takes care of provisioning or decommissioning any resources if you choose to apply the plan.
Terraform Plugins allow Terraform Core to communicate with your infrastructure host or SaaS providers. Terraform Providers and Provisioners are examples of plugins.
Some teams use Terraform for everything. The usage of Terraform provides many benefits over a custom-built CLI:
Centralized state management The ability to plan to see changes before they're applied GitHub pull request integration Native IDE support Ability to create modules Pull parameters from different secure mechanisms It creates a single tool for managing all the infrastructure, from an EC2 instance to the connectors installed, to the configuration deployed.
What is the value of Airbyte’s Terraform Provider? Programmatically manage Airbyte Airbyte resources can be created and managed through code rather than the UI. Interface with Airbyte from the development tools you use daily. Create new connections or quickly test configuration changes without impacting your Airbyte workspace used for Production.
Integrate your Airbyte configuration within your existing infrastructure. Changes to your Airbyte configuration can be proposed and reviewed by your entire team. Speed up your workflow by deploying changes to Airbyte as easily as merging a new pull request in GitHub.
Automation at scale Some teams desire to make configuration changes to Airbyte through automation alone rather than humans (who can be prone to human error). These teams can leverage the Terraform provider to interact with Airbyte programmatically, enabling them to manage Airbyte at scale and without human error. This automation can help reduce errors and improve efficiency, freeing up time for data engineers to focus on other tasks that move their business forward.
Use Case Example We have been leveraging our Terraform provider internally at Airbyte.
It has enabled us to:
Spin up brand new Airbyte connectors, and connections for testing This enabled us to quickly test configuration changes or new connectors without impacting our Production Airbyte workspace. Quickly update connector configurations using code instead of the UI. This enabled us to update our workspace and multiple connections simultaneously. Source control enables us to track our Airbtye configuration changes over time. “The Terraform provider allows us to modify multiple workspaces and connections quickly, saving us time and hundreds of clicks in the UI. We track these changes in source control and can roll back to previous configurations if needed. This gives us the peace of mind needed in a production system.” - Alex Gronemeyer, Lead Data Engineer, Airbyte
Availability The Terraform provider (and Airbyte API) is to be available for Airbyte Cloud, Airbyte OSS and self-hosted Airbyte Enterprise deployments.
Steps for Getting Started Today Check out our terraform setup guide with code examples. Navigate to the Airbyte Developer Portal . Within the portal, generate and save your Airbyte API key. Next, head to the Airbyte Terraform Provider page and select the button in the top-right corner titled "Use Provider" Copy and paste the source code into a file titled 'main.tf' Run: terraform init Run: terraform plan Run: terraform apply
