SFTP to Databricks: How to Move Your Files

Moving files from SFTP to Databricks is the plumbing behind a lot of enterprise data work. A partner drops a nightly CSV, a bank posts a settlement file, a legacy system exports Parquet on a schedule, and none of it is useful until it lands in a table someone can query.

This guide covers building that pipeline with Airbyte: which of the two SFTP connectors to use, what Databricks needs configured before the destination will connect, and the choice between parsing files into records and copying them across untouched.

SFTP to Databricks at a glance:

Why move files from SFTP into Databricks?

SFTP persists because it is boring and universal. Partners, banks, healthcare clearinghouses, and older ERPs all speak it, and none of them are going to expose an API on your schedule. The result is a server accumulating files that nobody can analyse without downloading them first.

Databricks solves the other half. Once those files are Delta tables in Unity Catalog, they are queryable in SQL, governed alongside everything else, joinable against your warehouse data, and available to notebooks and ML jobs. The work is in the middle: getting files off the server reliably, on a schedule, without re-reading everything each night.

Which SFTP connector should you use?

Airbyte ships two, and picking the wrong one costs you a rebuild. The plain SFTP source handles CSV and JSON only. The SFTP Bulk source is the one to reach for in almost every case: it consolidates many files into a single stream, supports incremental loading based on when files were created or modified, and can be told to load only the most recent file in a folder, which helps when a partner posts a full snapshot every night.

SFTP Bulk also handles more formats: CSV, JSONL, Parquet, Avro, and document types. The rest of this guide uses it.

How do you set up an SFTP to Databricks pipeline?

Step 1: Prepare SFTP authentication

You need a host address, a username, and either a password or an SSH key pair. For key authentication, generate the pair with ssh-keygen and copy the public key to the server with ssh-copy-id. Recent connector versions accept Ed25519, ECDSA, and DSS keys alongside RSA.

One detail trips people up constantly: the private key must be uploaded as a PEM file, not pasted into the field. Save it with the BEGIN and END lines included and no stray quotes or trailing spaces, as ssh.pem and use the upload control. On macOS or Linux, run chmod 600 ssh.pem first.

Step 2: Configure the SFTP Bulk source

Enter the host, username, and credentials, then define your streams. Each stream needs a name, a file type, and usually a glob pattern narrowing which files belong to it. Patterns work the way you would expect: include the folder for files in a subdirectory, and use a double asterisk to match recursively.

Two optional fields do real work. Folder Path scopes the search to one directory instead of the server root, which matters on a server holding years of archives. Start Date sets the floor for incremental syncs, so a first run does not pull a decade of history you do not want. The port defaults to 22.

Step 3: Set up Databricks for Airbyte

Unity Catalog is not optional, since the destination requires it. You also need a SQL warehouse or all-purpose cluster to run queries against, and from its Connection Details tab you collect the Server Hostname, HTTP Path, and Port. The catalog name is the top-level name in the workspace sidebar, not a schema or table.

Grant the connector permission to create schemas, tables, and Unity Catalog Volumes. That last one surprises people: Airbyte stages Avro files in Volumes before loading them into tables, so a principal that can write tables but not create Volumes will fail. OAuth2 with a service principal is the recommended authentication; a personal access token also works.

Step 4: Add the destination and run

Create the Databricks Lakehouse destination, enter the connection details and catalog name, accept the JDBC driver terms, and set a default schema. From version 4.0.0 the destination uses Direct Load: each stream is written straight to a final Delta table, and raw tables are no longer produced. Upgrading from 3.x is a breaking change covered in the migration guide. Connect source to destination, pick a sync mode, set a frequency, and run.

Should you parse the files or copy them raw?

This is the decision most people make by accident. The delivery method controls it.

Replicate Records parses structured files into individual records and writes them as typed table columns. This is what you want when the files are CSV, JSONL, Parquet, or Avro and the goal is queryable tables. Copy Raw Files transfers bytes exactly as they appeared on the server, preserving the original files without interpreting them. That is the right choice for unstructured or binary content, or when you need the original artifact retained for audit.

The two are not interchangeable. Replicate Records only works on structured formats the connector can parse, and it is not a workaround for large files: the 1.5 GB per-file ceiling applies either way. If your partner sends a 4 GB export, split it before upload.

What limits should you plan around?

• No deduplication from this source. SFTP Bulk supports full refresh and incremental append, but not incremental append with dedup. If the same record appears in two files, both rows land. Deduplicate downstream in Databricks rather than expecting the pipeline to handle it.
• Nested structures arrive as JSON strings. Airbyte object and array types map to Databricks STRING, serialized as JSON, rather than to native STRUCT or ARRAY columns. Nested Parquet or JSONL will need parsing in SQL after landing.
• Numbers become DECIMAL(38, 10). That allows 28 integer digits and 10 fractional. Values needing more fractional precision than that will not survive intact, which is worth checking against financial files before you trust the output.
• Time-only types become strings. Databricks has no native equivalent for time with or without timezone, so both land as STRING.

Schema and table names are lowercased automatically, though column names keep the casing from the source. Worth knowing before you write queries against what you expect the table to be called.

Frequently asked questions

Which SFTP connector should I use for multiple files?

SFTP Bulk. It consolidates many files into a single stream and supports incremental loading based on file creation or modification time. The standard SFTP source is limited to CSV and JSON and lacks bulk handling.

Do I need Unity Catalog to use the Databricks destination?

Yes. Unity Catalog is a hard prerequisite, and the connector also needs permission to create Volumes inside it, because Avro files are staged there before being loaded into tables.

How large a file can Airbyte move from SFTP?

Individual files are capped at 1.5 GB when using the Copy Raw Files delivery method. Switching to Replicate Records does not raise the ceiling. Split larger files before they reach the server.

Will Airbyte pick up new files automatically?

Yes, with incremental sync. Files added or modified after the configured Start Date are collected on subsequent runs, so you are not re-reading the whole directory each night.

Can I keep the original files rather than parsing them?

Yes, choose the Copy Raw Files delivery method. Bytes are copied exactly as they appeared on the server, which suits unstructured or binary content and cases where the original artifact must be retained.

Get your SFTP files into Databricks

Once Unity Catalog permissions are sorted, an SFTP to Databricks connection is quick to build and quietly collects new files from then on. Working with files in object storage instead of on a server? See our guide to Parquet to PostgreSQL for the same pattern against a different pair.

Connector behaviour described here reflects the SFTP Bulk source v1.9.x and Databricks Lakehouse destination v4.0.x, verified August 2026. Check the linked documentation for current versions.

Start syncing now →