Moving Files and Records together in Airbyte

Airbyte lets you move structured and unstructured data n a single connection.

COMPANY UPDATES

July 16, 2025

Charles Giardina

Summarize with AI:

Moving Files and Records together in Airbyte

AI is only as good as the data it ingests. Yet most data platforms force an artificial separation between structured records and unstructured files, creating gaps that AI models struggle to bridge. Our open, modular architecture makes it effortless for teams to move data no matter the situation, including the complex requirements of AI workloads.

Airbyte enables you to move structured records and unstructured files together in the same connection.

Let’s look at a scenario where you want to convert the data in your customer support environment into an S3 storage bucket that prepares it for ingestion by an LLM.

Transforming support articles into AI-ready knowledge

Say your support team operates on Zendesk Support. Your agents, noticing patterns, create knowledge articles that guide your customers through common problems. In reality, though, this data isn’t simple. Your agents add screenshots, videos, sample files, and more to these articles. These files enter Zendesk as attachments that can’t easily be integrated into Zendesk’s structured data. The result is much more complicated than traditional structured data. What’s in these files? What format are they? What do they explain? How big are they? When were they updated? What article do they belong to?

As you move data into your S3 destination, it might be tempting to ignore these unstructured files as miscellaneous content, but they’re actually critical to support cases. Without them, Zendesk’s structured data has far less meaning. Moving these files separately and later trying to connect them back to the articles they belong to seems equally unappealing. Fortunately, Airbyte is equipped to handle this situation for you. It moves your structured data and unstructured data in the same connection, along with structured metadata to describe your unstructured data.

Setting up your source

Start by setting up a source connector for Zendesk Support. You can authenticate with OAuth or an API token.

‍

When you’re ready, click Test and Save.

Setting up your destination

The next step is to set up your S3 destination, if you haven’t already. You’ll need these things to proceed:

A Role ARN, or an Access Key and Secret Access Key, from AWS.
Your S3 bucket name (for example, art-vandelay-imports-exports)
The directory you want to write to in that S3 bucket (for example, zendesk/support)
The AWS region your bucket is in (for example, us-east-2)

You can choose one of the following output formats: JSON Lines, CSV, Apache Avro, or Parquet. Your choice here governs not just the format of your structured data, but also the metadata Airbyte generates to describe your unstructured data.

‍

When you’re ready, click Test and Save.

Setting up your connection

With your source and destination ready to go, the next step is to create a connection. This is where Airbyte really shines.

Files are “just another stream”, meaning you can work with them like any stream, turning them on or off as needed, and modifying the schema as you go. In the screenshot of the schema below, all our article attachments live in the article_attachments stream.

You can also customize the metadata Airbyte generates to describe these attachments as if it were any ordinary field. Expand the article-attachments stream and customize which metadata fields you want to include.

When you’re ready, click Next, then click Finish & Sync.

‍

Once the sync completes, you can see the result in your S3 bucket. In the article_attachments stream, you’ll see two folders. One with your files, and another with your metadata in the JSON Lines format we chose earlier.

‍

Here’s an example of a screenshot from your articles, described in your JSONL file.


{
  "_airbyte_raw_id": "3a649fef-8099-4e9a-8807-be4092dcbd5a",
  "_airbyte_extracted_at": 1751474879696,
  "_airbyte_meta": {
    "sync_id": 41246493,
    "changes": []
  },
  "_airbyte_generation_id": 1,
  "_airbyte_data": {
    "id": 12138758717583,
    "url": "https://example.zendesk.com/api/v2/help_center/articles/attachments/12138758717583",
    "article_id": 12138789487375,
    "display_file_name": "Screenshot of web app showing the main features.png",
    "file_name": "Screenshot of web app showing the main features.png",
    "content_url": "https://example.zendesk.com/hc/article_attachments/12138758717583",
    "relative_path": "/hc/article_attachments/12138758717583",
    "content_type": "image/webp",
    "size": 109284,
    "inline": true,
    "created_at": "2025-03-11T23:33:57Z",
    "updated_at": "2025-03-11T23:33:57Z",
    "_airbyte_file_path": "/vandelay-imports-exports-article_attachments/2025_07_02_1751474877743_/article_attachments/12138758717583/Screenshot of web app showing the main features.png"
  }
}

That’s all there is to it! In just a few minutes, we synced unstructured files, qualified by metadata, to your destination in the same connection as your structured data.

File transfers for all

Zendesk is a natural example for moving structured and unstructured data together, but it’s far from the only one. Many API sources in the world have endpoints that return files. Airbyte’s built-in extensibility means access to file transfers is available for any data source. If you build your own connectors, you can use Airbyte’s YAML-based, low-code connector development kit to build file syncing into your own connectors without ever leaving Airbyte’s UI.

About the Author

Charles Giardina

VP of Engineering at Airbyte

Try Airbyte Agents

Be among the first to explore our new platform and get access to our latest features.

Try it free

Moving Files and Records together in Airbyte

Try Airbyte Agents

Related posts