n8n to Databricks: How to Sync Your Data

Syncing n8n to Databricks puts your workflow execution history into a governed lakehouse, alongside the rest of your operational data and within reach of notebooks, SQL, and ML jobs. It is a short build once two prerequisites are sorted, and one of them catches almost everyone.

This guide covers what the n8n source exposes, why its full refresh limitation decides your sync mode, and the Unity Catalog permission that is easy to miss.

n8n to Databricks at a glance:

What can you sync from n8n?

One stream: executions. The n8n source reads from the n8n API and exposes the record of workflow runs. Workflow definitions, credentials, users, and tags are not available through this connector.

Databricks is a good fit for that data when your automation history needs to sit under the same governance as everything else. Unity Catalog gives you lineage, access control, and a single catalog covering both execution telemetry and the business data those workflows touch. If the only goal is a failure-rate dashboard, a lighter destination will do; the case for Databricks is governance and proximity to other datasets.

Why does full refresh decide your sync mode?

The n8n source supports full refresh only. Every sync pulls every execution record from scratch, with no cursor and no incremental option.

That makes full refresh overwrite the right choice. Pairing a full-refresh source with append writes the whole execution history again on every run, so the Delta table grows by its own size each sync and you end up deduplicating in SQL forever. Overwrite replaces the table and leaves a clean snapshot.

It also means sync frequency has a real cost. A busy n8n instance accumulates executions quickly, and each run transfers all of them again through the staging layer. Daily, or a few times a day, is usually right. Hourly is rarely worth it for data that has not changed.

One more thing worth stating plainly: the n8n source carries a low sync success rate in Airbyte's own connector metadata. Build monitoring in rather than assuming that no alert means no problem.

How do you set up the pipeline?

Step 1: Create an n8n API key

In n8n, open Settings, then API, and select Create an API key. Configure the source with that key plus your instance host. Self-hosted and cloud instances both work. If your organisation restricts traffic by IP, add the Airbyte Cloud IP addresses to the relevant allow lists.

Step 2: Prepare Databricks

Unity Catalog is not optional, since the destination requires it. You also need a SQL warehouse or all-purpose cluster, and from its Connection Details tab you collect the Server Hostname, HTTP Path, and Port. The catalog name is the top-level name in the workspace sidebar, not a schema or table.

Now the part people miss. Grant the connector permission to create schemas, tables, and Unity Catalog Volumes. Airbyte stages Avro files in Volumes before loading them into Delta tables, so a service principal that can write tables but cannot create Volumes will fail even though its table permissions look correct. OAuth2 with a service principal is the recommended authentication; a personal access token also works.

Step 3: Add the destination and run

Create the Databricks Lakehouse destination, enter the connection details and catalog name, accept the JDBC driver terms, and set a default schema. From version 4.0.0 the destination uses Direct Load, writing each stream straight to a final Delta table with no raw tables. Upgrading from 3.x is a breaking change covered in the migration guide. Connect the source, select the executions stream, choose full refresh overwrite, set a frequency, and sync.

How do execution records land in Delta tables?

Mostly as you would expect, with a few mappings worth knowing before you write queries.

• Nested structures become JSON strings. Airbyte object and array types map to Databricks STRING, serialized as JSON, rather than to native STRUCT or ARRAY columns. Execution records carry nested node output, so expect to parse that in SQL after landing.
• Numbers become DECIMAL(38, 10). That allows 28 integer digits and 10 fractional, comfortably more than execution metadata needs.
• Time-only types become strings. Databricks has no native equivalent for time with or without timezone, so both land as STRING. Full timestamps are unaffected, which is what matters for computing durations.

Schema and table names are lowercased automatically, though column names keep their source casing. Check what was actually created before writing queries against the name you expected.

Frequently asked questions

Can I sync n8n workflows rather than executions?

No. The connector exposes a single executions stream. Workflow definitions, credentials, and users are not available. A custom connector built against the n8n API is the route if you need them.

Do I need Unity Catalog?

Yes. It is a hard prerequisite, and the connector also needs permission to create Volumes inside it, because Avro files are staged there before loading into Delta tables.

Why does my sync fail despite correct table permissions?

Check whether the principal can create Unity Catalog Volumes. Table-level permissions alone are not enough, since staging happens in Volumes before any table is written.

Which sync mode should I use?

Full refresh overwrite. The source cannot do incremental, so append would rewrite the entire execution history on every run and leave you with compounding duplicates.

Why is nested execution data a string rather than a struct?

The destination maps object and array types to STRING containing serialized JSON. Parse it in SQL with Databricks JSON functions when you need structured access to node output.

Get your n8n data into Databricks

Sort the Volumes permission first, pick overwrite, and set a cadence you can justify. Want a store built for aggregating this data instead? See our guide to n8n to ClickHouse. Moving files into the same lakehouse? See SFTP to Databricks.

Start syncing now →