Connect S3 Glue data with any data source.

Send data from every Airbyte source to S3 Glue.

Load data into S3 Glue Read the docs

Sources: 590+
Standard support: 0.1.11

Overview

What is the Airbyte S3 Glue connector?

The Airbyte S3 Glue connector loads data from 590+ Airbyte sources into S3 Glue on a schedule you control.

01
Load from any source
Move data into S3 Glue from 590+ Airbyte sources on a schedule you control.
02
Cloud or self-hosted
Run the S3 Glue connector on OSS.

Setup

Live in ten steps

Log in to your Airbyte account and navigate to the "Destinations" tab.
Click on "Add Destination" and select "S3 Glue" from the list of available connectors.
Enter your AWS access key ID and secret access key in the appropriate fields.
Select the AWS region where you want to store your data.
Enter the name of the S3 bucket where you want to store your data.
Choose the format in which you want to store your data (e.g. CSV, JSON, Parquet).
Enter the prefix for your data files (e.g. "mydata/").
Choose whether you want to compress your data files (e.g. Gzip).
Click on "Test Connection" to ensure that Airbyte can connect to your S3 Glue destination.
If the connection is successful, click on "Save" to save your S3 Glue destination configuration.

Configuration fields

S3 Glue configuration fields
Field	Type	Description
format*	object	Format of the data output. See here for more details
glue_database*	string	Name of the glue database for creating the tables, leave blank if no integration
glue_serialization_library*	string	The library that your query engine will use for reading and writing data in your lake.
s3_bucket_name*	string	The name of the S3 bucket. Read more here.
s3_bucket_path*	string	Directory under the S3 bucket where data will be written. Read more here
s3_bucket_region*	string	The region of the S3 bucket. See here for all region codes.
access_key_id	string	The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here.
file_name_pattern	string	The pattern allows you to set the file-name format for the S3 staging file(s)

Hybrid deployment

Sovereign by architecture, not by policy.

The data plane runs in your environment. Records, credentials and keys stay with you; the control plane only receives configuration and job metadata.

Your environment

DataConnectorsCredentialsKeys

↕ metadata only

Airbyte control plane

OrchestrationMonitoringUI

FAQ

S3 Glue connector questions

What data can I load into S3 Glue with Airbyte?

Airbyte moves data from any supported source into S3 Glue on a schedule you control, with full refresh and incremental sync where the destination supports it.

Does the S3 Glue connector support incremental sync?

Check the Airbyte docs for the sync modes the S3 Glue destination supports.

Where can I run the S3 Glue connector?

Run the S3 Glue connector on OSS.

Didn't find your answer? Talk to sales