Connectors

/

Destinations

/

Google Cloud Storage (GCS)

  • Destination
  • Warehouses and Lakes
  • Beta

Connect Google Cloud Storage (GCS) data with any data source.

Send data from every Airbyte source to Google Cloud Storage (GCS).

Sources
590+
Standard support
1.0.0

Overview

What is the Airbyte Google Cloud Storage (GCS) connector?

The Airbyte Google Cloud Storage (GCS) connector loads data from 590+ Airbyte sources into Google Cloud Storage (GCS), with full refresh and incremental sync on a schedule you control.

  • 01

    Load from any source

    Move data into Google Cloud Storage (GCS) from 590+ Airbyte sources on a schedule you control.

  • 02

    Incremental syncs

    Pull only the records that changed since the last run instead of reloading everything.

  • 03

    Cloud or self-hosted

    Run the Google Cloud Storage (GCS) connector on Cloud, OSS, Hybrid.

Setup

Live in five steps

  1. Create a GCS bucket, or select an existing bucket. The connection check writes, lists, and deletes a probe object. Ensure that retention policies and object holds permit this deletion.

  2. Create a dedicated service account for Airbyte and create an HMAC key for it. If your organization uses the constraints/storage.restrictAuthTypes organization policy, it must allow HMAC authentication. Save the HMAC secret when you create the key because Google Cloud doesn't provide it again.

  3. Grant the service account the Storage Object User role (roles/storage.objectUser) on the bucket. If you use a custom role, include these permissions: text storage.multipartUploads.abort storage.multipartUploads.create storage.objects.create storage.objects.delete storage.objects.list Delete access is required because the connection check deletes its probe object and Full Refresh - Overwrite syncs delete existing objects under the stream's output prefix.

Configuration fields

Google Cloud Storage (GCS) configuration fields
FieldTypeDescription
credential*objectAn HMAC key is a type of credential and can be associated with a service account or a user account in Cloud Storage. Read more here.
format*objectFormat of the data output. See here for more details
gcs_bucket_name*stringThe name of the GCS bucket. Read more here.
gcs_bucket_path*stringDirectory under the GCS bucket where data will be written.
file_name_patternstringPattern for output file names in the bucket directory. Available variables: {date} (yyyy_MM_dd), {date:yyyy_MM}, {timestamp}, {part_number}, {sync_id}, {format_extension}.
gcs_bucket_regionstringSelect a Region of the GCS Bucket. Read more here.
gcs_path_formatstringFormat string on how data will be organized inside the GCS bucket directory. Available variables: ${NAMESPACE}, ${STREAM_NAME}, ${YEAR}, ${MONTH}, ${DAY}, ${HOUR}, ${MINUTE}, ${SECOND}, ${MILLISECOND}, ${EPOCH}, ${UUID}, ${SYNC_ID}.

Hybrid deployment

Sovereign by architecture, not by policy.

The data plane runs in your environment. Records, credentials and keys stay with you; the control plane only receives configuration and job metadata.

  • Cloud
  • OSS
  • Hybrid

Your environment

DataConnectorsCredentialsKeys

↕ metadata only

Airbyte control plane

OrchestrationMonitoringUI

FAQ

Google Cloud Storage (GCS) connector questions

What data can I load into Google Cloud Storage (GCS) with Airbyte?

Airbyte moves data from any supported source into Google Cloud Storage (GCS) on a schedule you control, with full refresh and incremental sync where the destination supports it.

Does the Google Cloud Storage (GCS) connector support incremental sync?

Yes. The Google Cloud Storage (GCS) destination supports full refresh and incremental sync, so Airbyte can load only the records that changed since the last sync.

Where can I run the Google Cloud Storage (GCS) connector?

Run the Google Cloud Storage (GCS) connector on Cloud, OSS, Hybrid. Hybrid deployments keep the data plane in your environment while the control plane orchestrates syncs.

Didn't find your answer? Talk to sales

Start loading data into Google Cloud Storage (GCS) in minutes.

Free 14-day trial. No credit card needed.