How to get your file data (CSV, JSON, Excel, Parquet) ready for agentic AI
Load a CSV, JSON, Excel, or Parquet file from HTTPS, S3, GCS, Azure, or SFTP into your warehouse with Airbyte so AI agents query it as a governed table.

Airbyte's File source reads a single CSV, JSON, Excel, Feather, Parquet, YAML, or fixed-width file from HTTPS, cloud storage, or SFTP and loads it into your warehouse as a table that agents can query.
Plenty of useful data never lives in an application. Price lists, partner feeds, territory maps, budget files, and public datasets arrive as files. They get emailed, dropped on an SFTP server, or published at a URL. Agents asked about pricing, territories, or targets need that data, and they need it in the same place as everything else.
Handing an agent a raw file is fragile. It has to guess the delimiter, the header, and the types, and it has no way to join the file with other tables. Loading the file into a warehouse table fixes that: the parsing is decided once, the columns are typed, and the result sits next to CRM, billing, and product data under the same access controls.
Why file data matters for agentic AI
- It fills gaps. Reference and planning data often exists only as a file, so agents without it give incomplete answers.
- It comes from outside. Partner and vendor feeds, public datasets, and exports from tools with no API all arrive as files.
- It is messy. Tabs instead of commas, missing headers, date columns stored as text. Parsing rules need to be explicit.
- It changes in place. A file at the same URL is often replaced with a new version, so a scheduled reload keeps agents current.
What agents can do with file data
- Lookups against reference tables. Map a customer's postal code to a sales territory or an account to a price tier from a loaded file.
- Plan versus actual. Compare actual spend or revenue in the warehouse with targets loaded from a budget spreadsheet.
- Enrichment with public data. Join internal records with public datasets loaded from an HTTPS URL.
- Partner feed checks. Summarize what changed in a vendor's latest file after each scheduled reload.
How Airbyte gets file data ready
One file, one table
The File source replicates one file into one table. You set a Dataset Name, which becomes the table name in the destination, using letters, numbers, dashes, and underscores. Folders and glob patterns are not supported, so each file needs its own source. For whole folders in object storage, look at Airbyte's storage-specific sources instead.
Storage providers
- HTTPS (public web), with an optional User-Agent header
- Google Cloud Storage, with a service account JSON for private buckets
- Amazon S3, with access keys for private buckets
- Azure Blob Storage, with a SAS token or shared key for private storage
- SFTP, SSH, and SCP, with host, user, and optional password and port
- Local filesystem, on self-managed Airbyte only. Locally stored files can't be used in Airbyte Cloud.
For Google Drive, use the download URL format https://drive.google.com/uc?export=download&id=<file id>.
Formats and compression
Supported formats are CSV, JSON/JSONL, Excel, Excel Binary Workbook, fixed-width files, Feather, Parquet, and YAML. HTML, XML, and Pickle are not supported. Unstructured files such as raw text, audio, or video are not supported either.
Gzip and Zip compression are supported. Bzip2, LZMA, XZ, and Snappy are not.
JSON support is still experimental. Simple JSON schemas work, but files with several layers of nesting may not be handled well. If you need nested JSON, test it before relying on it.
Reader Options
Parsing is based on Pandas IO tools, and Reader Options pass a JSON object to the matching Pandas reader. For a tab-separated file with no header:
{ "sep": "\t", "header": null, "names": ["column1", "column2"], "parse_dates": ["column2"] }If a file has no header, set header to null, or the first record is lost. To control types, use dtype, for example {"dtype": "string"} to read every column as a string. For JSON files, options such as {"orient": "records"} change how the data is read.
For large remote files the connector uses the smart_open library, and you can switch to GCSFS or S3FS with the reader implementation setting.
Sync modes
The File source supports Full Refresh only. Each sync rereads the whole file. Incremental syncs and incremental deletes are not supported. For a file that is replaced at the same location, schedule the sync to match how often the file changes, and use Full Refresh Overwrite so the table mirrors the latest version.
Tips for reliable file loads
- Pin the parsing. Set the separator, header, and key column types in Reader Options so a new version of the file doesn't change how it is read.
- Keep the URL stable. Replace the file at the same location instead of publishing a new URL, so the same source keeps working.
- Name the table clearly. The Dataset Name is what agents see, so choose one that describes the content, such as
sales_territories.
How agents reach file data
The File source is not listed as an agent connector, so the context layer doesn't read files directly. Two paths work today:
- Sync to BigQuery or Snowflake. In Airbyte Cloud, the context layer can give agents read-only SQL access to those destinations, limited to
SELECTandSHOW TABLES. - Use the Airbyte MCP for pipeline work. At
https://mcp.airbyte.com/mcp, agents can create a File source, run a sync, and check job logs when a file fails to parse. Add?read-only=trueto limit an agent to read-only tools.
What this unlocks
- Reference, planning, and partner data available to agents as typed tables.
- Parsing rules set once in Reader Options instead of guessed per question.
- Scheduled reloads that keep the table matched to the latest file.
- Files joined with application, CRM, and billing data.
- Governed access to file data through the warehouse.
Getting started
- Host the file on HTTPS, S3, GCS, Azure Blob Storage, or SFTP.
- Create a File source with a Dataset Name, format, and URL.
- Add Reader Options for delimiters, headers, dates, and types.
- Sync to BigQuery or Snowflake with Full Refresh Overwrite, then connect your agent to the Airbyte MCP.
Try Airbyte free or talk to sales about file data for agents.
Frequently asked questions
Which file formats does the Airbyte File source support?
CSV, JSON and JSONL, Excel, Excel Binary Workbook, fixed-width files, Feather, Parquet, and YAML. HTML, XML, and Pickle are not supported, and neither are unstructured files such as raw text, audio, or video.
Can the File source read a folder of files or a glob pattern?
No. The File source replicates one file at a time into one table, named by the Dataset Name you set. Folders and glob patterns are not supported.
Does the File source support incremental syncs?
No. It supports Full Refresh only, so each sync rereads the whole file. Incremental syncs and incremental deletes are not supported.
How do I control column types or parsing?
Use Reader Options, a JSON string passed to the underlying Pandas reader. For CSV you can set the separator, header, column names, date columns, and dtype, for example forcing all columns to strings.
Can I use a file stored on my laptop with Airbyte Cloud?
No. Locally stored files can't be used as a source in Airbyte Cloud. The local filesystem option is available only on self-managed Airbyte. On Cloud, host the file on HTTPS, S3, GCS, Azure Blob Storage, or an SFTP server.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.
