Build an open data lakehouse with Dremio and Airbyte

Name: Airbyte — Build an open data lakehouse with Dremio and Airbyte
Author: Airbyte

Learn how to move all your data to a data lake and connect your data lake with the Dremio lakehouse platform.

Intermediate

Jeremiah Morrow

updatedAugust 11, 2022

connectors used

Dremio

Airbyte

Once a connection with your Amazon S3 data lake has been established, you can promote your dataset to a Physical Dataset in Dremio. Physical Datasets (PDS) are entities that represent variously formatted data as tables. These entities are called physical datasets because they represent actual data. You can create a PDS from one data file or from a folder that contains data files.

Navigate to the parquet file(s) in the bucket you connected Airbyte to. Click the “Format File” icon on the right of your file(s).

Dremio will recognize this as a Parquet file, and it will create the PDS.

Click Save and Dremio will show a preview of the table.

You can save this as a Virtual Dataset, also known as a View. Virtual datasets are not copies of data, so they use very little memory, and they always reflect the current state of the parent datasets they are derived from.

Now that you have a dataset in Dremio, you can connect to your BI or data science tools like Tableau, Power BI, Jupyter notebooks, and many more.

Get started with Airbyte and Dremio

As you have seen in this tutorial, Dremio and Airbyte make it very easy and intuitive to get data from a variety of sources into the data lake, and start creating dashboards and visualizations. It takes just a few minutes to create new data sources with Airbyte, and once all your data is in the data lake, Dremio makes it easily available to data consumers for analytics.

Airbyte Cloud provides a free trial to get started centralizing your data into a data lake. If you’re ready to get started with an easy and open data lakehouse, you can sign up for your forever-free tier of Dremio Cloud. You can check out the getting Started with Dremio guide to learn more about the additional features and functionality of Dremio.

Build agents on live data

The Free plan includes 1,000 Agent Operations and 50+ agent connectors.

Try it free