Article

Orchestrate your Airbyte ELT Jobs with Dagster

•

February 10, 2022

•

3 min read

Summarize with ChatGPT

In 2021, Airbyte has been focusing at expanding the number of connectors, but also the number of integrations within the data stack, in order to enable any data teams to fully leverage Airbyte’s ELT features. Since its inception 18 months ago, Airbyte has been used by more than 12,000 companies to sync data and it now counts more than 170 connectors. Since our Series-B in December, our vision to commoditize data integration has only been reinforced. This includes the full integration of Airbyte with the data stack. In a step to help data teams have better control over their data pipelines, Airbyte announces today its partnership and integration with Dagster. By doing so, Airbyte closes the loop with the integration of popular open-source orchestration platforms: Airflow, Prefect and now Dagster.

Dagster is an orchestration platform for the development, production, and observation of data assets that is an open-source project maintained by Elementl. Dagster enables practitioners to define jobs in terms of the data flowwith a fast development and testing workflow. Dagster unifies disparate data tools into a cohesive data platform, making practitioners dramatically more productive, scaling from “single-player-mode” to large organizations, and providing unparalleled context for all stakeholders.

Both Airbyte and Dagster open-source platforms are passionate about helping data engineers, analysts and scientists have better leverage of their time, by giving them better control over their data flows. This partnership is another step in this direction.

"Airbyte is an emerging open-source standard for data ingestion with a ton of momentum. With their focus on openness, fast development, it was an ideal partnership. We're were thrilled to work with them to build a best-in-class experience for the modern data practitioner looking to make Airbyte work seamlessly in production," said Nick Schrock, CEO of Elementl.

The integration with Dagster enables organizations using Airbyte to leverage Dagster’s orchestration platform to trigger Airbyte syncing jobs. Airbyte users now have full control over how they operate ELT jobs, along with their other data workflows and automations. The integration also enables these data teams to have better visibility over the sync failures and therefore better resolution to automations that might follow a sync job.

Given Airbyte’s ambition to address the long tail of connectors with a goal of reaching 500+ connectors by the end of 2022, Dagster users will be able to benefit from this growing list of high-quality maintenance-free connectors, thus saving more and more engineering time. Airbyte’s open-source and non-opinionated nature gives data teams a lot more flexibility and control over their pipelines, in comparison to other ELT platforms.

You can read the following technical tutorial on how to leverage the Airbyte-Dagster integration - give it a shot!

A note: we love purple octopus, and Octavia couldn’t resist!

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 30-day free trial