Connect S3 data with any data source.

Replicate any source into S3 without custom pipelines.

Sources: 590+
Certified support: 1.9.10

Overview

What is the Airbyte S3 connector?

The Airbyte S3 connector loads data from 590+ Airbyte sources into S3, with full refresh and incremental sync on a schedule you control.

01
Load from any source
Move data into S3 from 590+ Airbyte sources on a schedule you control.
02
Incremental syncs
Pull only the records that changed since the last run instead of reloading everything.
03
File transfer
Move files through the pipeline as-is, without parsing them into records first.
04
Cloud or self-hosted
Run the S3 connector on Cloud, OSS, Hybrid.

Setup

Live in twelve steps

Log into your Airbyte Cloud account.
In the left navigation bar, click Destinations. In the top-right corner, click + new destination.
On the destination setup page, select S3 from the Destination type dropdown and enter a name for this connector.
Configure fields: - Access Key Id - See this on how to generate an access key. - We recommend creating an Airbyte-specific user. This user will require read and write permissions to objects in the bucket. - Secret Access Key - Corresponding key to the above key id. - Role ARN - See this on how to create a role. - S3 Bucket Name - See this to create an S3 bucket. - S3 Bucket Path - Subdirectory under the bucket to sync the data into. - S3 Bucket Region: - See here for all region codes. - S3 Path Format - Additional string format on how to store data under S3 Bucket Path. Default value is ${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_. - S3 Endpoint - Leave empty if using AWS S3, fill in S3 URL if using Minio S3. - S3 Filename pattern - The pattern allows you to set the file-name format for the S3 staging file(s), next placeholders combinations are currently supported: {date}, {date:yyyy_MM}, {timestamp}, {part_number}, {sync_id}, {format_extension}. Please, don't use empty space and not supportable placeholders, as they won't be recognized.
Click Set up destination.
Go to local Airbyte page.
In the left navigation bar, click Destinations. In the top-right corner, click + new destination.
On the destination setup page, select S3 from the Destination type dropdown and enter a name for this connector.
Configure fields: - Access Key Id - See this on how to generate an access key. - See this on how to create a instanceprofile. - We recommend creating an Airbyte-specific user. This user will require read and write permissions to objects in the staging bucket. - If the Access Key and Secret Access Key are not provided, the authentication will rely either on the Role ARN using STS Assume Role or on the instanceprofile. - Secret Access Key - Corresponding key to the above key id. - Make sure your S3 bucket is accessible from the machine running Airbyte. - This depends on your networking setup. - You can check AWS S3 documentation with a tutorial on how to properly configure your S3's access here. - If you use instance profile authentication, make sure the role has permission to read/write on the bucket. - The easiest way to verify if Airbyte is able to connect to your S3 bucket is via the check connection tool in the UI. - S3 Bucket Name - See this to create an S3 bucket. - S3 Bucket Path - Subdirectory under the above bucket to sync the data into. - S3 Bucket Region - See here for all region codes. - S3 Path Format - Additional string format on how to store data under S3 Bucket Path. Default value is ${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_. - S3 Endpoint - Leave empty if using AWS S3, fill in S3 URL if using Minio S3. - S3 Filename pattern - The pattern allows you to set the file-name format for the S3 staging file(s), next placeholders combinations are currently supported: {date}, {date:yyyy_MM}, {timestamp}, {part_number}, {sync_id}, {format_extension}. - Please, don't use empty space and not supportable placeholders, as they won't be recognized.
Click Set up destination.
Each stream has its own directory.
The data output files can be sorted by upload time.

Configuration fields

S3 configuration fields
Field	Type	Description
format*	object	Format of the data output. See here for more details
s3_bucket_name*	string	The name of the S3 bucket. Read more here.
s3_bucket_path*	string	Directory under the S3 bucket where data will be written. Read more here
s3_bucket_region*	string	The region of the S3 bucket. See here for all region codes.
access_key_id	string	The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here.
file_name_pattern	string	Pattern to match file names in the bucket directory. Read more here
role_arn	string	The ARN of the AWS role to assume. Only usable in Airbyte Cloud.
s3_endpoint	string	Your S3 endpoint url. Read more here

Hybrid deployment

Sovereign by architecture, not by policy.

The data plane runs in your environment. Records, credentials and keys stay with you; the control plane only receives configuration and job metadata.

Cloud
OSS
Hybrid

Your environment

DataConnectorsCredentialsKeys

↕ metadata only

Airbyte control plane

OrchestrationMonitoringUI

FAQ

S3 connector questions

What data can I load into S3 with Airbyte?

Airbyte moves data from any supported source into S3 on a schedule you control, with full refresh and incremental sync where the destination supports it.

Does the S3 connector support incremental sync?

Yes. The S3 destination supports full refresh and incremental sync, so Airbyte can load only the records that changed since the last sync.

Where can I run the S3 connector?

Run the S3 connector on Cloud, OSS, Hybrid. Hybrid deployments keep the data plane in your environment while the control plane orchestrates syncs.

Didn't find your answer? Talk to sales