- Destination
- Warehouses and Lakes
- Alpha
Connect S3 Data Lake data with any data source.
Load data from any Airbyte source into S3 Data Lake on the schedule you choose.
- Sources
- 590+
- Certified support
- 0.4.1
Overview
What is the Airbyte S3 Data Lake connector?
The Airbyte S3 Data Lake connector loads data from 590+ Airbyte sources into S3 Data Lake, with full refresh and incremental sync on a schedule you control.
- 01
Load from any source
Move data into S3 Data Lake from 590+ Airbyte sources on a schedule you control.
- 02
Incremental syncs
Pull only the records that changed since the last run instead of reloading everything.
- 03
Cloud or self-hosted
Run the S3 Data Lake connector on OSS.
Setup
Live in three steps
Confirm your S3 Data Lake destination credentials and permissions.
Configure the destination in Airbyte with the required connection settings.
Choose a source, map streams, and schedule syncs into S3 Data Lake.
Configuration fields
| Field | Type | Description |
|---|---|---|
| catalog_type* | object | Specifies the type of Iceberg catalog (e.g., NESSIE, GLUE, REST, POLARIS) and its associated configuration. |
| main_branch_name* | string | The primary or default branch name in the catalog. Most query engines will use "main" by default. See Iceberg documentation for more information. |
| s3_bucket_name* | string | The name of the S3 bucket that will host the Iceberg data. |
| s3_bucket_region* | string | The region of the S3 bucket. See here for all region codes. |
| warehouse_location* | string | The root location of the data warehouse used by the Iceberg catalog. Typically includes a bucket name and path within that bucket. For AWS Glue and Nessie, must include the storage protocol (such as "s3://" for Amazon S3). |
| access_key_id | string | The AWS Access Key ID with permissions for S3 and Glue operations. |
| flush_batch_size_mb | integer | The approximate size in megabytes of each batch of data written to Iceberg. Smaller values flush more frequently, improving data freshness and reducing data loss on failure, but will create more small files that require compaction. Must be between 1 and 500 MB. Default is 200 MB. |
| normalize_column_names | boolean | Lowercases column names and replaces non-alphanumeric characters with underscores before writing to Iceberg (e.g. "Foo.Bar" -> "foo_bar"). Enable this when downstream query engines cannot handle mixed-case or special-character column names. IMPORTANT: toggling this on existing synced streams requires clearing the stream data and running a full refresh. |
Hybrid deployment
Sovereign by architecture, not by policy.
The data plane runs in your environment. Records, credentials and keys stay with you; the control plane only receives configuration and job metadata.
- OSS
Your environment
Airbyte control plane
Related connectors
FAQ
S3 Data Lake connector questions
What data can I load into S3 Data Lake with Airbyte?
Airbyte moves data from any supported source into S3 Data Lake on a schedule you control, with full refresh and incremental sync where the destination supports it.
Does the S3 Data Lake connector support incremental sync?
Yes. The S3 Data Lake destination supports full refresh and incremental sync, so Airbyte can load only the records that changed since the last sync.
Where can I run the S3 Data Lake connector?
Run the S3 Data Lake connector on OSS.
Didn't find your answer? Talk to sales
Start loading data into S3 Data Lake in minutes.
Free 14-day trial. No credit card needed.