Connectors

/

BigQuery

BigQuery

Warehouses and Lakes

Sync BigQuery data anywhere.

BigQuery is a cloud-based data warehousing and analytics platform that allows users to store, manage, and analyze large amounts of data in real-time. It is a fully managed service that eliminates the need for users to manage their own infrastructure, and it offers a range of features such as SQL querying, machine learning, and data visualization. BigQuery is designed to handle petabyte-scale datasets and can be used for a variety of use cases, including business intelligence, data exploration, and predictive analytics. It is a powerful tool for organizations looking to gain insights from their data and make data-driven decisions.

  • Standard
  • Alpha
Sync BigQuery dataRead the docs
  • 50+Destinations
One connector

Everything BigQuery can do in Airbyte

  • Sync to your warehouse

    Land BigQuery data in 50+ destinations on a schedule you control.

  • Incremental syncs

    Pull only the records that changed since the last run instead of reloading everything.

  • One authorization

    Authenticate BigQuery once and Airbyte keeps every scheduled sync running on it.

  • Cloud or self-hosted

    Run the BigQuery connector on Cloud, Self-Managed Enterprise.

Sync capabilities

  • Full Refresh SyncSupported
  • Incremental SyncSupported
  • Available onCloud, Self-Managed Enterprise
  • Destinations50+ Airbyte connectors
  • Connector version0.4.5

Set up in 10 steps

  1. First, you need to have a Google Cloud Platform account and a project with BigQuery enabled.
  2. Go to the Google Cloud Console and create a new service account with the necessary permissions to access your BigQuery data.
  3. Download the JSON key file for the service account and keep it safe.
  4. Open Airbyte and go to the Sources page.
  5. Click on the "Create a new source" button and select "BigQuery" from the list of available sources.
  6. Enter a name for your source and click on "Next".
  7. In the "Connection Configuration" section, enter the following information: - Project ID: the ID of your Google Cloud Platform project - JSON Key: copy and paste the contents of the JSON key file you downloaded earlier - Dataset: the name of the dataset you want to connect to
  8. Click on "Test Connection" to make sure everything is working correctly.
  9. If the test is successful, click on "Create Source" to save your configuration.
  10. You can now use your BigQuery source connector to extract data from your dataset and load it into Airbyte for further processing.
Authentication

Authenticate BigQuery once

  • Credentials JSON

    The contents of your Service Account Key JSON file. See the docs for more information on how to obtain this key.

Explore
All connectors
FAQ

Common questions

What is ETL?

ETL, an acronym for Extract, Transform, Load, is a vital data integration process. It involves extracting data from diverse sources, transforming it into a usable format, and loading it into a database, data warehouse or data lake. This process enables meaningful data analysis, enhancing business intelligence.

What data can you extract from BigQuery?

BigQuery provides access to a wide range of data types, including:

1. Structured data: This includes data that is organized into tables with defined columns and data types, such as CSV, JSON, and Avro files.
2. Semi-structured data: This includes data that has some structure, but not necessarily a fixed schema, such as XML and JSON files.
3. Unstructured data: This includes data that has no predefined structure, such as text, images, and videos.
4. Time-series data: This includes data that is organized by time, such as stock prices, weather data, and sensor readings.
5. Geospatial data: This includes data that is related to geographic locations, such as maps, GPS coordinates, and spatial databases.
6. Machine learning data: This includes data that is used to train machine learning models, such as labeled datasets and feature vectors.
7. Streaming data: This includes data that is generated in real-time, such as social media feeds, IoT sensor data, and log files.

Overall, BigQuery's API provides access to a wide range of data types, making it a powerful tool for data analysis and machine learning.

How do I transfer data from BigQuery?

1. First, you need to have a Google Cloud Platform account and a project with BigQuery enabled.

2. Go to the Google Cloud Console and create a new service account with the necessary permissions to access your BigQuery data.

3. Download the JSON key file for the service account and keep it safe.

4. Open Airbyte and go to the Sources page.

5. Click on the "Create a new source" button and select "BigQuery" from the list of available sources.

6. Enter a name for your source and click on "Next".

7. In the "Connection Configuration" section, enter the following information:
- Project ID: the ID of your Google Cloud Platform project
- JSON Key: copy and paste the contents of the JSON key file you downloaded earlier
- Dataset: the name of the dataset you want to connect to

8. Click on "Test Connection" to make sure everything is working correctly.

9. If the test is successful, click on "Create Source" to save your configuration.

10. You can now use your BigQuery source connector to extract data from your dataset and load it into Airbyte for further processing.

What are top ETL tools to transfer data from BigQuery?

The most prominent ETL tools to transfer data to include: Airbyte, Fivetran, StitchData, Matillion, Talend Data Integration. These tools help in extracting data from various sources (APIs, databases, and more), transforming it efficiently, and loading it into and other databases, data warehouses and data lakes, enhancing data management capabilities.

What is ELT?

ELT, standing for Extract, Load, Transform, is a modern take on the traditional ETL data integration process. In ELT, data is first extracted from various sources, loaded directly into a data warehouse, and then transformed. This approach enhances data processing speed, analytical flexibility and autonomy.

Difference between ETL and ELT?

ETL and ELT are critical data integration strategies with key differences. ETL (Extract, Transform, Load) transforms data before loading, ideal for structured data. In contrast, ELT (Extract, Load, Transform) loads data before transformation, perfect for processing large, diverse data sets in modern data warehouses. ELT is becoming the new standard as it offers a lot more flexibility and autonomy to data analysts.

Start moving BigQuery data today

Free for 14 days on Airbyte Cloud. Set up the BigQuery connector once and let Airbyte keep it in sync.

Try it freeTalk to sales