How to get your S3 data ready for agentic AI
Land source data in Amazon S3 as Parquet, Avro, CSV, or JSONL with Airbyte, or as Iceberg tables, so AI agents and query engines read a governed copy.

Airbyte writes source data to Amazon S3 as Parquet, Avro, CSV, or JSONL files, or as Iceberg tables with the S3 Data Lake destination, giving agents and query engines a governed copy in your own bucket.
Not every team wants agents querying a warehouse. Some keep their analytical data in object storage and use query engines on top. Others need a copy of source data in a bucket they control for compliance, cost, or to feed several tools at once. Amazon S3 is the usual home for that copy.
For agents, a data lake works when the files are predictable: a consistent layout, a typed format, and clear rules about when files are replaced. This guide covers how Airbyte writes to S3 so that holds, and how the S3 Data Lake destination adds table semantics with Apache Iceberg.
Why S3 data matters for agentic AI
An S3 copy of your source data has a few properties that suit agent workloads:
- You own the storage. Data sits in your AWS account under your IAM policies and bucket settings.
- Many readers. The same files can serve query engines, notebooks, and retrieval pipelines that agents depend on.
- Open formats. Parquet, Avro, and Iceberg are widely supported, so you are not tied to one engine.
- Cost control. Object storage is a cheap place to keep history that agents only occasionally need.
What landing data in S3 means for agents
- Predictable paths. Each stream is written to its own directory, which works like a table. The default path format is
${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, and you can customize it with variables such as${HOUR},${UUID}, and${SYNC_ID}. - Typed, columnar files. Parquet output keeps types and lets engines read only the columns a query needs.
- Sized files. A sync can produce multiple files, partitioned by size with a target of about 200 MB compressed.
- Generation tagging. Each object is tagged with
x-amz-meta-ab-generation-id, so you can tell which sync generation wrote it.
How Airbyte gets S3 data ready
The S3 destination
Formats. Avro, CSV, JSONL, and Parquet.
- Parquet supports several compression codecs, including Snappy and Zstandard.
- Avro supports deflate, bzip2, xz, Zstandard, and Snappy.
- CSV can flatten the record into columns, and JSONL can be GZIP compressed.
Sync modes.
- Full Refresh Overwrite
- Full Refresh Append
- Incremental Append
Deduplicated modes are not supported. In Full Refresh Overwrite, files from earlier generations are deleted after a successful sync. If a sync fails partway, files from more than one generation may remain until the next successful sync. Airbyte recommends a dedicated bucket or prefix for this reason.
Authentication.
- IAM user access key and secret.
- IAM role assumed through STS with a role ARN and external ID. On Airbyte Cloud, a member of the Airbyte team must enable this, so contact sales to use it.
- Instance profile for self-managed deployments running on EC2 or EKS.
The destination also works with S3-compatible storage such as MinIO by setting the endpoint.
The S3 Data Lake destination (Iceberg)
When agents need tables rather than files, the S3 Data Lake destination writes Apache Iceberg tables to S3 or S3-compatible storage. It is newer than the S3 destination, so check its release stage in the connector registry before relying on it in production.
- Catalogs: REST, AWS Glue, Nessie, and Polaris.
- Sync modes: all five, including Incremental Append + Deduped, so tables can hold one current row per key.
- Types: top-level fields map to Iceberg types. Objects and unions are written as JSON strings, and arrays keep their structure when the source declares an item type. Timestamps with time zones are converted to UTC.
- Schema evolution: the docs describe which changes are handled automatically, and recommend Full Refresh Overwrite to absorb unsupported changes.
- Authentication: IAM user in most cases, or an IAM role on Airbyte Cloud with the Glue catalog.
Layout choices that help agents
A few configuration decisions make the bucket easier for agents and their tools to read:
- Keep one connection per prefix. Because Full Refresh Overwrite deletes earlier generations under the stream's path, sharing a prefix between connections or other writers risks deleting files you meant to keep.
- Add a date or hour to the path when agents usually ask about recent periods. Engines that understand path partitions can then skip older files.
- Prefer Incremental Append for large sources. It adds new files with only the new data, so history accumulates without rewriting everything on each sync.
- Use a stream prefix on the connection when several sources share a bucket, so directory names stay unambiguous.
How agents reach S3 data
The Airbyte context layer's read-only SQL access is documented for BigQuery and Snowflake destinations, not for S3. Agents reach S3 data through the tools you put on top of the bucket, such as a query engine that reads Parquet or Iceberg.
Airbyte still helps agents with the pipeline itself. Through the Airbyte MCP at https://mcp.airbyte.com/mcp, an agent can create a connection to an S3 destination, run syncs, check sync status, and read job logs. Use ?read-only=true if the agent should only observe.
If your agents also need direct reads or search, you can send the same sources to BigQuery or Snowflake in a second connection and keep S3 as the long-term copy.
What this unlocks
- A governed copy of source data in your own AWS account.
- Typed Parquet files or Iceberg tables that agents' query engines can read.
- Predictable paths and generation tags for auditing.
- Deduplicated tables on S3 with the Iceberg destination.
- Pipeline management by agents through the Airbyte MCP.
Getting started
- Create a dedicated bucket or prefix and an IAM policy for Airbyte.
- Choose the S3 destination with Parquet output, or S3 Data Lake with your Iceberg catalog.
- Connect your sources with Incremental Append, or Append + Deduped on Iceberg.
- Point your query engine at the bucket and connect your agent to it and to the Airbyte MCP.
Try Airbyte free or talk to sales about data lakes for agents.
Frequently asked questions
Which file format should I use for agent workloads on S3?
Parquet is usually the best choice because it is columnar and typed, so query engines read only the columns a query needs. The S3 destination also writes Avro, CSV, and JSONL, with compression options for each.
Does the S3 destination deduplicate records?
No. The S3 destination supports Full Refresh Overwrite, Full Refresh Append, and Incremental Append, but not the deduplicated modes. If you need one current row per key, use the S3 Data Lake destination, which writes Iceberg tables and supports Incremental Append + Deduped.
What happens to old files in Full Refresh Overwrite mode?
After a successful sync, files from previous generations are deleted and the current generation is kept. Each object is tagged with x-amz-meta-ab-generation-id. Airbyte recommends a dedicated bucket or prefix to avoid deleting unrelated data.
Can agents query S3 directly through the Airbyte context layer?
Not at the time of writing. The context layer's read-only SQL access is documented for BigQuery and Snowflake destinations. Agents read S3 data through a query engine you run over the bucket, or you can also sync the same sources to a warehouse.
How does the S3 destination authenticate?
With IAM user access keys, an IAM role assumed through STS, or an instance profile on self-managed deployments. IAM role authentication on Airbyte Cloud must be enabled by the Airbyte team, so contact sales to use it.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.
