Amazon Redshift to Databricks: How to Move Your Data

Moving Amazon Redshift into Databricks is a warehouse to lakehouse pipeline, and unlike most of the pairings in this series it has no awkward mismatch to work around. Both systems are built for analytical data at scale, both handle wide tables well, and nothing needs flattening or reshaping on the way through.

That makes this an unusually straightforward build, and it shifts the interesting questions elsewhere: what the cursor-based source can and cannot see, and what reading a large Redshift table does to the cluster your analysts are also using. This guide covers the managed path with Airbyte.

Amazon Redshift to Databricks at a glance:

Why move data from Redshift to Databricks?

Two situations account for most of these pipelines.

The first is capability rather than storage. Redshift is a good warehouse and a limited environment for machine learning, notebooks and Python. Teams that need those bring the data to Databricks rather than trying to make Redshift into something it is not, and the two coexist for years.

The second is a migration, where Databricks is becoming the analytical platform and Redshift is being retired gradually rather than in one weekend. The pipeline keeps the lakehouse current while dashboards and models move across, and the cursor decisions below matter more in that case because the copy is on its way to becoming the original.

What do you need before you start?

Fewer prerequisites than most pipelines in this series, which is itself worth noting:

A read-only Redshift user. Scoped to the schema you are reading. The connector is built on the JDBC code base and does not alter the schema present in your warehouse, so it is genuinely a read-only operation on the source side. The Redshift source documentation covers the connection fields.

A cursor column on every table you sync incrementally. An updated_at or equivalent that only increases and is set on every write. For incremental dedupe you also nominate primary key columns. Tables without one are full refresh, which on a warehouse table is an expensive read rather than an impossible one.

A sync window that avoids your busy period. Redshift concurrency is finite, and a sync reading large tables competes with the analytical queries the cluster exists to serve. This is a scheduling decision rather than a configuration one, and it is the most common cause of complaints about this pipeline.

Unity Catalog access. Each stream becomes a table in Unity Catalog and the connector needs permission to create Volumes for Avro staging. Agree the catalog and schema with whoever owns governance before the first sync.

If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list on the Redshift cluster before you begin.

How do you build a Redshift to Databricks pipeline in Airbyte?

Step 1: Audit your tables for cursors

Go through the tables you intend to replicate and check which have a column that only increases and is set on every write. Tables built by your own transformation jobs usually do or easily can. Tables loaded by a third party may not, and those are the ones that will be full refresh whether you like it or not, so identify them now and size the schedule around them.

Step 2: Configure the Redshift source

Click Sources in the left navigation, then New Source, and select Redshift, following adding a source. Supply the host, port, database, schema and credentials. The check connection tool in the interface is the quickest way to confirm your security group and IP rules are right before you configure anything else.

Step 3: Configure the Databricks destination

Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply your workspace details, the catalog and schema, and authentication. Airbyte creates one table per stream with its own metadata columns alongside your data, which is worth mentioning to whoever reviews the resulting schema.

Step 4: Create the connection and schedule off-peak

Click Connections, then New connection, select your tables, nominate cursor fields, and pick a sync mode per stream. Then set the schedule to run shortly after your overnight models finish rather than during the working day, which avoids competing for cluster concurrency at exactly the time analysts are using it.

Start with a handful of tables and compare row counts against Redshift before extending. On a warehouse to warehouse pipeline the numbers are the whole point, and a discrepancy found on five tables is a morning while one found on ninety is a project.

What does the cursor miss?

Incremental sync here is cursor-based, and the connector extracts changes with a query that amounts to selecting everything where the cursor is greater than its value at the last sync. That is simple and reliable, and it has two blind spots worth stating explicitly.

A row modified without its cursor column being updated will not be picked up. On tables your own models rebuild this is within your control, which is why the audit in step 1 matters: if a transformation rewrites rows without touching a timestamp, the lakehouse will quietly hold stale values while every sync reports success.

And deletions are invisible, because a query filtered on a cursor cannot observe an absence. If rows are removed in Redshift they persist in Databricks indefinitely. Where that matters, a soft-delete flag in the source table filtered downstream is the clean fix and a periodic full refresh is the blunt one. On a migration in progress, resolve this before anyone treats the lakehouse copy as authoritative.

What does this do to your Redshift cluster?

It consumes concurrency like any other query, and that is the practical constraint on this pipeline rather than anything to do with the destination. Redshift allocates a finite number of concurrent slots, and a sync reading several large tables occupies some of them for as long as it runs.

The failure mode is not an error, it is analysts noticing that queries are slower on Tuesday mornings. That makes it hard to attribute, which is why scheduling deliberately matters more than it sounds. Running after your overnight transformations complete puts the load in a window where the cluster is otherwise idle.

Two other habits reduce it. Sync narrow tables or views rather than wide raw ones, since Redshift is columnar and a query touching three columns of two hundred is cheap. And split very large tables into their own connection with a less frequent schedule, so one heavy table does not dictate the cadence of everything else.

Frequently asked questions

Does the Redshift source support change data capture?

No. Incremental sync is cursor-based and needs a user-defined cursor field such as an updated_at column. Deletions are not captured.

Why are some updated rows not appearing in Databricks?

The cursor column was not updated when those rows changed. The connector selects records where the cursor exceeds its last value, so a modification that leaves the cursor untouched is invisible.

Will this slow down my Redshift cluster?

It consumes concurrency like any other query. Schedule syncs outside your busy analytical window, ideally after overnight models complete, and read narrow tables rather than wide raw ones.

Is there a volume limit on this pipeline?

Not in the way there is with a relational destination. A lakehouse is built for analytical volume, so the constraint here is what reading the data costs on the Redshift side rather than what the destination can hold.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. You may want SQL views in Redshift to narrow wide tables before syncing, which is short and reduces cluster load.

Get your Redshift data into Databricks

Audit your tables for cursors first, since that decides which are incremental and which are expensive. Schedule outside your busy window, split heavy tables into their own connection, and resolve what a deleted row should mean before anyone treats the Databricks copy as authoritative.

Airbyte's connector catalog includes 700+ pre-built connectors, so the same lakehouse can hold warehouse data alongside operational sources. For Redshift into an operational database instead, see Amazon Redshift to PostgreSQL, and for another columnar source landing in Databricks, ClickHouse to Databricks.

Start syncing now →