Article

Announcing Airbyte Self-Managed Enterprise: The Engine for Self-Serve Data Platforms

•

September 24, 2024

•

5 min read

Summarize with ChatGPT

As your data infrastructure needs to evolve, closed-source ETL/ELT solutions often fall short. You’ll likely have found that these solutions struggle to support both internal compliance must-haves, as well as the full range of data sources your business needs visibility into. This lack of flexibility leads to the development of bespoke pipelines—custom, error-prone, hard to maintain solutions built to bridge the gaps in your data architecture. This fragmentation can stifle innovation and slow down the pace of your data operations.

The only way to future-proof your data needs is to build on open technologies. Airbyte Self-Managed Enterprise is designed to eliminate these challenges by providing a robust, flexible, and scalable data ingestion solution that can grow with your business - without critical data ever leaving your firewall.

Introducing Airbyte Self-Managed Enterprise in General Availability

Starting today, we’re excited to announce the general availability of Airbyte Self-Managed Enterprise, providing your business with best-in-class data access while ensuring robust governance and control over its movement.

By combining centralized control of user access with Airbyte’s self-serve data ingestion capabilities, Self-Managed Enterprise becomes an engine for multi-tenant data mesh or data fabric architectures. Perhaps most importantly, Airbyte Self-Managed Enterprise allows you to retain full control over your sensitive data. With a fully self-managed solution that can be operated in air gapped environments via UI, API and Terraform SDK, you can ensure that no data ever leaves your environment.

Airbyte Self-Managed Enterprise introduces several new capabilities targeted towards large organizations designed to enhance your data platform’s capabilities and security:

Multitenancy & Role-Based Access: Manage multiple teams and projects within a single Airbyte deployment. This feature empowers citizen developers to discover and consume data across your organization, all while managing team access from a single pane of glass.

PII Masking: Protect sensitive information by hashing personally identifiable information (PII) as it moves through your pipelines. This ensures compliance with internal company policies, privacy regulations and allows for greater pipeline consolidation within Airbyte.

Certified Enterprise Source Connectors: Available exclusively to Enterprise customers, these connectors—starting with Oracle and Workday—are designed for production-readiness and come with the heightened support to expect from all of Airbyte’s certified connectors.

Enterprise Support with SLAs: Airbyte Self-Managed Enterprise comes with dedicated support and guaranteed service level agreements (SLAs), ensuring that your data movement infrastructure remains reliable and performant, and expert assistance is available when needed.

To support at-scale organizations with hundreds of concurrent pipelines, Self-Managed Enterprise is deployed on Kubernetes in the environment of your choice, with certified support for Amazon Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS) and Google Kubernetes Engine (GKE).

Example: Self-Serve Data Platforms with Self-Managed Enterprise on AWS

In this section, we provide an example technical overview of Airbyte Self-Managed Enterprise as the data infrastructure layer in a data platform built on AWS. Airbyte supports out-of-the-box the following scale and resilience capabilities, which you can see in the following implementation diagram:

Simplified view of Airbyte’s Internal Implementation

Native to Kubernetes: Airbyte’s architecture is fully compatible with Kubernetes, allowing for scalable and resilient deployments. Kubernetes ensures that your data pipelines can scale horizontally, handle failovers seamlessly, and integrate effortlessly with other cloud-native tools.
CI/CD Integration: Airbyte offers seamless integration with your CI/CD pipelines through its Terraform Provider, API and language specific libraries, such as our Python SDK. This ensures auditability and proper review protocols for any changes to your ingestion layer. This integration also facilitates automated deployments and consistent environments, reducing human error and accelerating the deployment process.
Isolation between Control and Data Planes: In a simplified view of Airbyte, the control plane may run independently of Airbyte data planes. In the future, this will be used to enable Self-Managed Enterprise deployments orchestrated in one place, spanning across multiple regions or environments.

Detailed installation steps can be found in our implementation guide for Airbyte Self-Managed Enterprise, which walks you through production-ready, high availability deployments of Airbyte across all major cloud providers. In AWS, this includes external storage of Airbyte configuration in a Postgres database instance typically running in Amazon RDS or Amazon Aurora. It also extends to external S3 storage of connection state and sync logs, and external persistence of source and destination connector secrets in Amazon Secrets Manager, ensuring these are persisted in a secure environment.

As Airbyte is native to Kubernetes, you can easily deploy on helm or ArgoCD while keeping the Airbyte configuration all to Kubernetes secrets combined with a single values.yaml file. The skeleton configuration included below is all you would need to configure to get started with Self-Managed Enterprise:


global:
  edition: enterprise
  airbyteUrl: 
  auth:
    instanceAdmin:
      firstName: 
      lastName: 
    identityProvider:
      type: oidc
      oidc:
        domain:
        app-name: 
        display-name: 
  database:
    host:
    port:
    database:
    userSecretKey:
    passwordSecretKey:
  storage:
    type: "S3"
    bucket: 
      log: airbyte-bucket
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
    s3:
      region:
      authenticationType: instanceProfile
secretsManager:
  type: awsSecretManager
  awsSecretManager:
    region: 
    authenticationType: instanceProfile

If you are currently using Airbyte’s Open Source edition, upgrading to Airbyte Self-Managed Enterprise is a straightforward process. Just like all of Airbyte’s products, our enterprise edition builds on the foundation of the open-source platform and protocol.

Getting Started with Airbyte Self-Managed Enterprise

Are you building a data platform or modernizing your existing pipelines? Schedule a kickoff with our account and implementation teams to learn more about Airbyte’s Enterprise products and support packages, or sign up on our upcoming Self-Managed Enterprise webinar.

In the meantime, you can also check the other announcements of Airbyte 1.0:

Airbyte 1.0 is prime-time ready with a commitment to reliability and quality
Airbyte unlocks the long tail with the new Connector Marketplace and AI Assistant
How Airbyte 1.0 empowers GenAI and RAG workflows

The future of data integration is here, and it’s open!

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 30-day free trial

About the Author

Alex is a product manager at Airbyte, leading the company's enterprise offerings. Before joining Airbyte, he was at Datadog, where he developed observability products.