The difference between Airbyte and Airflow
Airbyte and Airflow are two popular tools that can be used for data integration, and because there is some overlap between these tools, they are sometimes confused with each other. At a high-level, Airbyte is a tool that moves data from one system to another. On the other hand, Airflow is an orchestrator tool that schedules and periodically executes a sequence of tasks in a particular order. In this article you will learn more about the focus of each of these tools, the main differences between them, and why Airflow is sometimes thought of as an ETL/ELT tool.
What is Airbyte?
Airbyte is an ELT tool that moves data from a source system to a destination system. To do this, it periodically executes a sync run which reads records from a source and sends the extracted records to a destination. One of the main benefits of Airbyte is that it can be used to easily extract data from hundreds of different sources, and load it in any of its many supported destinations.
It is possible and relatively common to specify how often to execute a sync run within Airbyte itself, i.e. do a sync every 24 hours. However, it is also possible to hand-off the scheduling of Airbyte sync runs to an orchestrator such as Airflow, Dagster, or Prefect. The reason for allowing an orchestrator to handle the scheduling of sync runs is to allow better coordination with other tasks that may be prerequisites for a sync run, or that may need to be executed upon the completion of a sync run. This will be discussed in more detail in the next section.
What is Airflow?
Airflow is often configured to execute a sequence of tasks on a schedule, and is particularly useful when the execution of some tasks are dependent on the completion of other tasks. To better understand this, let's imagine a hypothetical scenario, in which an Airbyte sync is one of the tasks in a sequence that is scheduled to be periodically executed by Airflow, as follows:
- A CRM system dumps data to S3.
- Airbyte reads the dumped data from S3, and sends it into a BigQuery destination
- After the data has been transmitted to BigQuery, a custom SQL transformation job is executed inside BigQuery, which combines the newly loaded data with other data that is already in BigQuery.
In this example, an orchestrator would be configured to do the following:
- Wait until the scheduled time to start the execution of the sequence. For example, it could be scheduled to execute every 24 hours.
- Trigger a dump of CRM data to S3.
- Wait until the data dump has fully completed
- Trigger Airbyte to sync the data from S3 to Bigquery.
- Wait for the Airbyte sync to complete.
- Trigger the transformation job.
- Wait for the transformation job to complete.
This sequence of tasks is demonstrated in the following image:
As you can see from the above sequence, Airbyte may just be one of many tasks that Airflow is responsible for scheduling and triggering!
Is Airflow an ETL and/or ELT tool?
Airflow provides built-in operators or hooks and community-managed operators or hooks which can be used to execute or trigger many different kinds of tasks, including external programs – this concept was demonstrated above with the example of Airflow triggering an Airbyte sync run.
However, Airflow is not a purpose-built ETL or ELT tool, it is an orchestration tool. But because Airflow can execute an arbitrary sequence of tasks, it can be used for orchestrating and scheduling a sequence of tasks which may include an Extract (E) task, a Load (L) task, and/or a Transform (T) task. When used for executing a series of tasks that perform ELT, you can think of Airflow as being responsible for triggering the execution of the Extract, the Load, and the Transform tasks in a sequence – but it is not necessarily responsible for executing these tasks by itself.
Airflow is not a purpose-built ELT tool. Therefore, if it is used for orchestrating the Extract (E), Load (L), and Transform (T) tasks required for executing an ELT pipeline, it is necessary to string the individual tasks together and to pass data between them. This can be time-consuming, complex, and error prone. On the other hand, using a tool such as Airbyte for ELT simplifies your ELT pipelines because it is purpose-built for the job. If you are interested in a deep dive into a discussion about using Airflow for ELT, check out the article on Airbyte’s blog called ETL Pipelines with Airflow: the Good, the Bad and the Ugly.
Conclusion
Airbyte and Airflow are tools that can help meet your data integration needs. In this article, you have learned about the similarities and differences between Airbyte and Airflow. Airbyte is used for moving data from one system to another. On the other hand, Airflow is an orchestrator tool that schedules and periodically executes a sequence of tasks in a particular order.
If you have enjoyed this article, you may be interested in other Airbyte tutorials, or in Airbyte’s blog. You may also consider joining the conversation on our community Slack Channel, participating in discussions on Airbyte’s discourse, or signing up for our newsletter.
