- Destination
- Warehouses and Lakes
- Beta
Connect Google Cloud Storage (GCS) data with any data source.
Send data from every Airbyte source to Google Cloud Storage (GCS).
- Sources
- 590+
- Standard support
- 1.0.0
Overview
What is the Airbyte Google Cloud Storage (GCS) connector?
The Airbyte Google Cloud Storage (GCS) connector loads data from 590+ Airbyte sources into Google Cloud Storage (GCS), with full refresh and incremental sync on a schedule you control.
- 01
Load from any source
Move data into Google Cloud Storage (GCS) from 590+ Airbyte sources on a schedule you control.
- 02
Incremental syncs
Pull only the records that changed since the last run instead of reloading everything.
- 03
Cloud or self-hosted
Run the Google Cloud Storage (GCS) connector on Cloud, OSS, Hybrid.
Setup
Live in five steps
Create a GCS bucket, or select an existing bucket. The connection check writes, lists, and deletes a probe object. Ensure that retention policies and object holds permit this deletion.
Create a dedicated service account for Airbyte and create an HMAC key for it. If your organization uses the constraints/storage.restrictAuthTypes organization policy, it must allow HMAC authentication. Save the HMAC secret when you create the key because Google Cloud doesn't provide it again.
Grant the service account the Storage Object User role (roles/storage.objectUser) on the bucket. If you use a custom role, include these permissions: text storage.multipartUploads.abort storage.multipartUploads.create storage.objects.create storage.objects.delete storage.objects.list Delete access is required because the connection check deletes its probe object and Full Refresh - Overwrite syncs delete existing objects under the stream's output prefix.
In Airbyte, enter the bucket name, bucket path, bucket region, HMAC access key, HMAC secret, and output format.
Test the connection. If your deployment restricts outbound network traffic, allow access to the GCS XML API endpoint at https://storage.googleapis.com.
Configuration fields
| Field | Type | Description |
|---|---|---|
| credential* | object | An HMAC key is a type of credential and can be associated with a service account or a user account in Cloud Storage. Read more here. |
| format* | object | Format of the data output. See here for more details |
| gcs_bucket_name* | string | The name of the GCS bucket. Read more here. |
| gcs_bucket_path* | string | Directory under the GCS bucket where data will be written. |
| file_name_pattern | string | Pattern for output file names in the bucket directory. Available variables: {date} (yyyy_MM_dd), {date:yyyy_MM}, {timestamp}, {part_number}, {sync_id}, {format_extension}. |
| gcs_bucket_region | string | Select a Region of the GCS Bucket. Read more here. |
| gcs_path_format | string | Format string on how data will be organized inside the GCS bucket directory. Available variables: ${NAMESPACE}, ${STREAM_NAME}, ${YEAR}, ${MONTH}, ${DAY}, ${HOUR}, ${MINUTE}, ${SECOND}, ${MILLISECOND}, ${EPOCH}, ${UUID}, ${SYNC_ID}. |
Hybrid deployment
Sovereign by architecture, not by policy.
The data plane runs in your environment. Records, credentials and keys stay with you; the control plane only receives configuration and job metadata.
- Cloud
- OSS
- Hybrid
Your environment
Airbyte control plane
FAQ
Google Cloud Storage (GCS) connector questions
What data can I load into Google Cloud Storage (GCS) with Airbyte?
Airbyte moves data from any supported source into Google Cloud Storage (GCS) on a schedule you control, with full refresh and incremental sync where the destination supports it.
Does the Google Cloud Storage (GCS) connector support incremental sync?
Yes. The Google Cloud Storage (GCS) destination supports full refresh and incremental sync, so Airbyte can load only the records that changed since the last sync.
Where can I run the Google Cloud Storage (GCS) connector?
Run the Google Cloud Storage (GCS) connector on Cloud, OSS, Hybrid. Hybrid deployments keep the data plane in your environment while the control plane orchestrates syncs.
Didn't find your answer? Talk to sales
Start loading data into Google Cloud Storage (GCS) in minutes.
Free 14-day trial. No credit card needed.