Announcing Prefect integration with Airbyte to automate ELT pipelines

Prefect Integration with Airbyte - Streamlining ELT pipelines with Prefect integration.

COMPANY UPDATES

December 10, 2021

3 min read

John Lafleur

Summarize with AI:

Announcing Prefect integration with Airbyte to automate ELT pipelines

Since its inception 16 months ago, Airbyte has been focusing at helping data teams replicate data from any source - starting with APIs, files, databases and warehouses - to any destinations - starting with data warehouses, lakes and databases. Airbyte has now been used by more than 9,000 companies to sync data. Airbyte now boasts more than 160 connectors, 55 of which were contributed by its growing community.

In a step to help data teams have better control over their data pipelines, Airbyte announces today its partnership and integration with Prefect. Prefect provides the leading dataflow automation platform that reduces engineering time and removes headaches for data engineers and data scientists by automating pipelines and workflows of any complexity. Prefect does so without ever seeing its customers' code to ensure privacy and security while providing visibility into what is running and where issues may arise.

Both Airbyte and Prefect’s open-source platforms are passionate about helping data engineers, analysts and scientists have better leverage of their time, by giving them better control over their data flows. This partnership is another step in this direction.

“Prefect has been one of our most requested integrations from our users since we started. Our growing community can now leverage Prefect’s platform to trigger automation workflows with their ELT jobs, including programmatically,”says Michel Tricot, co-founder and CEO of Airbyte.

The integration with Prefect enables organizations using Airbyte to leverage Prefect’s orchestration platform to trigger Airbyte syncing jobs. This grants Airbyte users full granular control over ELT job scheduling, empowering users to schedule Airbyte syncs to run exactly when they need to , along with their other data workflows and automations. The integration also enables these data teams to have better visibility over the sync failures and therefore better resolution to automations that might follow a sync job.

Given Airbyte’s ambition to address the long tail of connectors with a goal of reaching 500+ connectors by the end of 2022, Prefect users will be able to benefit from this growing list of high-quality maintenance-free connectors, thus saving more and more engineering time. Airbyte’s open-source and non-opinionated nature gives data teams a lot more flexibility and control over their pipelines, in comparison to other ELT platforms.

“Building integrations takes a lot of time, and as schemas and APIs change, dataflows can fail—and that means data downtime and late nights. The excitement around Airbyte’s approach to this problem in our community has been palpable,” said Andrew Black, Head of Partnerships at Prefect. “Data engineers love Airbyte’s extensibility, flexibility, and ease of use and are grateful to have an open source option. We think by using Prefect and Airbyte together, data engineers will be able to save even more time and further reduce headaches.”

You can check the Prefect Task documentation or read this recipe on how to build an ELT pipeline with Prefect, Airbyte and dbt - give it a shot!

About the Author

John Lafleur

John Lafleur is the COO and Co-Founder of Airbyte.