Article

What's new in Airbyte Platform | May 2025

•

May 13, 2025

•

8 min read

Summarize with ChatGPT

We’re excited to share our latest set of updates to the Airbyte Platform. Between full support for loading data to Apache Iceberg-compatible Data Lakes, AI Assistants for managing connection health, to full support for advanced APIs in the Connector Builder, we are continuing to double down on providing the best modern open data movement solution to data engineering teams. Here’s ten new features to help you build with confidence.

Consolidate Data to Iceberg-Compatible Data Lakes

Iceberg has quickly become a standard for building modern data platforms ready for providing AI-ready data to your teams. Our Iceberg-compatible Data Lake destination is catalog and storage agnostic, and designed for highly scalable and performant AI and analytics workloads. With schema evolution support, along with expanded capabilities to move unstructured data and structured records all in one pipeline, you can use Airbyte to consolidate on Iceberg with confidence knowing your data is AI ready. And, with Mappings, you can share corporate data with confidence, knowing sensitive data will not be leaked.

For a deep dive for data engineers on the benefits of adopting the Iceberg standard for storing both raw and processed data, and an outline of the capabilities of Airbyte's Data Lake destinations, check out this video:

‍

Operate Hundreds of Pipelines in One Place

As the number of pipelines you need to manage with Airbyte grows, the need to oversee, monitor and manage your data pipelines in one place is critical for maintaining high data quality and data freshness. With this in mind, we're excited to introduce four new capabilities enabling you to better manage hundreds of pipelines all in one place:

Diagnose sync errors with AI

We’ve expanded AI support in Cloud Team to allow you to quickly diagnose and fix failed data pipeline syncs Instantly analyze Airbyte logs, connector documentation and known issues to help you identify root cause, and get actionable solutions, without any manual debugging required. Read more here.

Monitor connection health from Connections page

Monitor the health of all your connections directly from within the Connections page using the new Connections Dashboard. This helps you quickly track down intermittent failures, and easily drill in for more information to help you resolve sync or performance issues.

‍

Organize pipelines with connection tags

Connection Tags help to visually group and organize your pipelines, making it easier than ever to find the connections you need. You can use tags to organize connections based on any set of criteria you like: 'department' in the case of different consuming teams, 'env' for indicating if they are running in production, and anything else you like.

‍

Identify schema changes in the Connection timeline

The Connection timeline now includes events for any connection settings update: whether these be a schedule update, or a change in the connection schema. For Cloud Teams users, you can use this in conjunction with AI logging to easily diagnose why sync behavior or volumes have suddenly changed.

‍

Manage Connectors as Infrastructure with Airbyte's Terraform Provider

Data movement is an integral part of your application and infrastructure. We've heard plenty of feedback from users requesting better ease of use for our Terraform Provider. We are excited to announce new capabilities making it easier than ever to manage all of your connectors as infrastructure using the Airbyte Terraform provider to roll out changes programmatically to your dev, staging, and production environments.

When building a connector in the Airbyte UI, you will now find a Copy JSON button at the bottom of connector configuration. This makes it easier than ever to export the the configuration of a connector to Terraform. This takes into account version-specific configuration settings, and can also be repurposed for configuring connectors with PyAirbyte, the Python SDK or the Airbyte API.

‍

Create custom connectors directly from YAML or Docker images

New endpoints and resources have also been added to the APIs and Terraform provider to allow you create and update custom connectors using a Connection Builder YAML manifest or Docker image. These endpoints do not allow you to modify Airbyte’s public connector configurations, but if you have custom endpoints within your organization and are running OSS or self-managed versions of Airbyte, these additional capabilities can be used to programmatically spin up new connectors for different environments.

If you need to manage API custom connectors in infrastructure, we now recommend you build your custom connector using the Connector Builder, test it using the in-app capability for verifying your connector, then export the configuration YAML. You can then easily pass in the YAML as part of a connector resource definition in Terraform:

Together, these two changes will make it significantly easier to manage your entire catalog of connectors as infrastructure in code, if this is preference for you and your team.

‍

Build Advanced API Connectors in Minutes with the Connector Builder

We're excited to announce new capabilities in the Connector Builder making it easier than ever to build and maintain complex API Connectors with minimal effort. Starting today, you can build connectors with OAuth 2.0 authentication patterns, endpoints kicking off asynchronous jobs, new response formats such as CSV, and finally those requiring GraphQL.

With this latest set of improvements, we feel like we've delivered on the core of our original vision for the Connector Builder. We're more than excited by the wide adoption of this capability by the community, which in the last 6 months has been used to introduce more than 100 new connectors to the Airbyte Marketplace.

Declarative 0Auth 2.0 support

No more manual sharing of tokens with Airbyte when building a connector which requires OAuth 2.0. You can now authenticate directly within the Connector Builder:

‍

Asynchronous streams

Many APIs respond with large data sets in stages, and data pipelines require request polling moving data in increments. Often, data is then downloaded from a different endpoint than the endpoint where the data was requested, making it even more complex to build a connector to this type of system. The new Asynchronous streams support in the Connector Builder handles all of this, and allows you to configure a connection where data extraction occurs over multiple steps.

‍

GraphQL request body support

You can now build connectors which require request bodies as GraphQL. This includes built-in syntax validation, and allows for the Connector Builder to support internal API endpoints on GraphQL you may need to extract data from:

‍

Handle CSV, gzip, and ZIP response formats

Airbyte now has two new ways to handle API responses in the Connector Builder.

CSV decoding: You may requests to sources that respond in CSV format. You can set expected delimiter and encoding options, ensuring data lands exactly as you need it to.
Nested decoding. You may now make requests to sources that respond with compressed gzip or zip files, then set a nested decoding option to handle the unzipped contents within.

‍

Summary

The Airbyte May Platform release is designed to help you build and manage your data pipelines efficiently and at scale. We’ve introduced ten new capabilities enabling you to better build and manage your data pipelines:

Consolidate Data to Iceberg-Compatible Data Lakes
Use in-app LLM Assistants diagnose connection health
Jump to faulty pipelines with hard-to-spot, intermittent failures
Filter down sets of connections to those you need with Connection Tags
See all connection settings updates in the Connection Timeline
Create custom connectors directly from YAML or Docker images
Declarative 0Auth 2.0 support
Asynchronous support
GraphQL request body support
Handle CSV, gzip, and Zip response formats

You can read more detailed information on all features available in our release note page, and stay up to date with the latest community events and activities by joining the Airbyte Community slack channel.

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 30-day free trial