Airbyte Winter Release 2024

Our 2024 Winter Release is full of exciting announcements that we wanted to share and celebrate!

COMPANY-UPDATES

February 28, 2024

Justin Chau

Summarize with AI:

Our mission at Airbyte is to make data available and actionable to everyone, everywhere. When we first released Airbyte at the end of 2020, we were not expecting the community to grow so fast nor Airbyte becoming a household name in the data & AI industry so quickly.

Now here we are with over 150,000 installations of Airbyte and a lot of pull from the community to communicate more about the progress of Airbyte. As such we are super proud to resume our Launch Week event (we actually only did one in the past in 2022), and we will make it a rendez-vous time for all our users every quarter from now on.

For this one we have pretty big releases we want to show you. The first one is about our new Airbyte library: PyAirbyte, which we believe is going to be a massive enabler for AI apps engineers and data engineers. The second is about our new Governance, Scale & Security features for Enterprise organizations.The third touches on our new way of building reliable and scalable destinations. And finally, we want to present all the performance improvements we’ve done in the platform and the push to connect to more unstructured data sources.

So without further ado, let’s dig into the details!

PyAirbyte: Airbyte “Pipelines-as-Code” - Public Beta

Until now, the value of Airbyte has been anchored in three pillars: Connectors, a robust platform orchestrating them, and intuitive interfaces. These elements are integrated into Airbyte Self-Managed, Airbyte Cloud, and Powered by Airbyte offerings.

Say hello to PyAirbyte (public beta)!

Elevating our commitment to innovation, we're thrilled to introduce PyAirbyte!

PyAirbyte is an open-source Python library that brings in the era of "Pipelines-as-Code", packaging our connectors and making them available as code, while removing the need for strict dependencies on Docker, Kubernetes, or an Airbyte Cloud account.

This latest addition to the Airbyte family represents a strategic evolution from our core offerings, catering to engineers who prefer a code-based approach to designing and managing data pipelines, in addition to our API and Terraform provider. It's designed to bridge the gap between the flexibility of custom Python scripts and the power of a data integration platform.

With PyAirbyte, Python users gain access to the majority of Airbyte’s data connectors – rather than having to build and maintain those themselves.

Getting started with PyAirbyte is as easy as installing the Python library:

pip install airbyte

Then, users can set up a source connector and begin extracting data in no time. Here's a quick peek at how it works:

import airbyte as ab

# Create and configure the source connector:
source = ab.get_source(
    "source-shopify",
    config={
        "shop": userdata.get("SHOP"),
        "credentials": {
            "auth_method": "api_password",
            "api_password": userdata.get("SHOP_API_PASSWORD")
        }
    }
)

# List the available streams available for the Shopify source
source.get_available_streams()

# Select the streams we are interested in loading to cache
source.select_streams(["products", "product_variants", "collections", "customers"])

# Read into DuckDB local default cache
cache = ab.get_default_cache()
result = source.read(cache=cache)


# Read from the cache into a pandas Dataframe:
products = cache.get_pandas_dataframe(stream_name="products")

The era of custom ETL scripts is over. With PyAirbyte, engineers can opt for a quicker return on investment and a more cost-effective solution than ever before.

If you’d like to know more about PyAirbyte, read our announcement blog post, or head straight to Quickstarts.

Security and Scale for Enterprise Organizations

The latest improvements to our enterprise Airbyte offerings (currently in early access) are all focused on improving the operational readiness of Airbyte at scale. As a first step, we’re rolling security and user management tooling, making it easier for you to securely roll out Airbyte across your entire organization of data engineers:

Multitenancy and Single Sign-On (SSO): You can now onboard multiple users onto a single self-managed Airbyte organization, and enable multiple engineers to simultaneously build pipelines on Airbyte without interference - all while keeping user access secure and managed from your identity provider.

Multiple Workspaces: To further support governance at scale, you can create and manage multiple isolated workspaces on a single Airbyte deployment. This makes it easier for different teams to work in Airbyte at once, all while ensuring a secure environment for moving data.

Role-Based Access Control (RBAC): Users in Self-Managed Enterprise all have roles, tied to specific workspaces. You can safely add new users with read-only access to workspaces, and onboard a new team to a set of isolated workspaces.

Application Key Management: Using the Airbyte API or our Terraform Provider to manage your Airbyte deployment is easier than ever. In Self-Managed Enterprise, application keys can be created and revoked all in one place, and are now tied to the associated user’s role.

And we’re just getting started. To explore these capabilities further, we encourage you to attend our upcoming demo on March 28th, or to speak with our sales team about Airbyte’s enterprise offerings, available both in self-managed and Airbyte Cloud.

Increasing Airbyte Connector Breadth

As you know, accessing correct data in a timely manner is no longer a luxury, it is a necessity. The frequency and complexity of decisions made with data continues to grow, and this requires constant access to an always growing list of data sources. No update from us is ever complete without new announcements on connector breadth.

On database connectors, we’ve been focused on improving our MongoDB and MySQL connector offerings - mostly towards better handling sync jobs at large scale with techniques such as checkpointing and chunking.

On destination connectors, we’ve significantly improved our mechanism for loading data with improved per-row error handling and incremental delivery of data for large syncs.

On AI, to ensure you have the data to power internal AI products, we’ve begun building unstructured data sources, and we’re now releasing Google Drive, OneDrive and SharePoint connectors. We’ve also developed a number of vector database destination connectors, which allow Airbyte to move data from hundreds of sources directly to your vector database destination (see our recent blog post on the ELTP paradigm for data pipelines).

You can learn more about AI use cases enabled by Airbyte via the following resources:

Conclusion

It is thanks to all of you, the community, for helping us achieve this vision. There is so much more in store to show you so stay tuned for the Spring release coming soon! You can also check out the SnakeByte game, and try to get the new high score while you wait. See you on the next one!

About the Author

Justin Chau

Developer Advocate at Airbyte