DynamoDB to Databricks: How to Move Your Data

Moving DynamoDB into Databricks means turning a schemaless key-value store into something Spark can model, and the interesting part happens before any data moves. DynamoDB tables do not have schemas. Lakehouse tables do. Something has to invent one, and that something is Airbyte's discovery phase.

This guide covers the managed path with Airbyte. Understanding how the schema gets inferred is the difference between a pipeline you trust and one that silently drops fields, and the lakehouse gives you somewhere useful to put the parts that do not fit.

DynamoDB to Databricks at a glance:

Why move data from DynamoDB to Databricks?

Two situations account for most of these pipelines.

The first is analysis DynamoDB cannot do. It is superb at fetching an item by its key and awkward at joins, aggregations and ad hoc filters, so anything shaped like how many or grouped by needs the data on separate compute built for it.

The second is machine learning and joining. Application data next to your other sources, in a form Spark and notebooks handle, under Unity Catalog governance. Unlike a relational destination there is no volume ceiling to design around, which matters because DynamoDB tables are chosen precisely when data gets large.

What do you need before you start?

Two credentials decisions and two that determine whether the tables are trustworthy:

AWS credentials, or better, an IAM role. Supply an access key and secret for user-based access, or omit both and the connector falls back to the default credentials provider, picking up the role attached to the workload running it. On self-managed Airbyte the second option avoids long-lived keys in configuration.

A cursor attribute that satisfies the rules. Incremental sync needs a top-level attribute, not one nested inside a map, of string or integer type, in ISO 8601 or epoch format, and never blank on any item. Single-table designs that bury timestamps inside nested attributes cannot use them without restructuring.

Knowledge of your sparse attributes. Discovery samples your table, so an attribute appearing only on items outside that sample will not reach Databricks. Ask your application developers what optional attributes exist rather than trusting the discovered schema.

Unity Catalog access. Each stream becomes a table in Unity Catalog and the connector needs permission to create Volumes for Avro staging. Agree the catalog and schema before the first sync.

If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a DynamoDB to Databricks pipeline in Airbyte?

Step 1: Establish what the items actually contain

Sit down with whoever owns the table and list the attributes, including the optional ones and any that changed type over the years. This takes twenty minutes and it is the only way to check the discovered schema against reality, because a schemaless store cannot tell you what it should contain.

Step 2: Configure the DynamoDB source

Click Sources in the left navigation, then New Source, and select DynamoDB, following adding a source. Supply the region and either your access keys or nothing at all if you are using a role. Then review the discovered schema against your list from step 1, and refresh it after any deploy that adds attributes.

Step 3: Configure the Databricks destination

Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply your workspace details, the catalog and schema, and authentication. Airbyte preserves the raw record alongside the typed columns, which on a schemaless source is the audit trail for what the pipeline actually received.

Step 4: Create the connection and plan for deletes

Click Connections, then New connection, select your tables and a sync mode. The source is cursor-based rather than reading DynamoDB Streams, so deletions never reach Databricks and updates are only seen when they advance the cursor attribute. Decide now whether that matters.

Where deletions genuinely occur and you cannot add a soft-delete flag, a periodic full refresh is the remedy. On DynamoDB that is a full scan consuming read capacity, so schedule it deliberately rather than nightly out of habit.

How does the schema get invented?

The connector scans your table with a limit of 1,000 items. If the returned data exceeds 1MB it scans again with the same limit, repeating until it has at least 1,000 items. It then combines every top-level attribute it saw and infers each attribute's type from its value.

Two consequences follow. An attribute appearing only on items outside that sample will not be in the catalogue: on a table where two percent of items carry an optional attribute, a sample drawn from the front may miss it entirely. And the connector assumes an attribute has the same type everywhere, which DynamoDB does not enforce, so a field that is a number on older items and a string on newer ones behaves unpredictably.

Neither is a bug, and the lakehouse is a forgiving place to land them. Because Airbyte keeps the raw record beside the typed columns, a field discovery missed is still physically present and can be recovered in Spark once you notice. That is the main advantage this destination has over a relational one for this source.

How should you model the result?

Treat the landed table as a bronze layer rather than an endpoint. Parse and cast into a silver table your analysts query, and keep the raw column so a schema surprise means reprocessing rather than starting from the day you noticed.

Anchor that table on DynamoDB's own keys. The partition key groups related items and the sort key orders within a partition, and together they are the natural identifier. Partition keys are usually meaningful too, since applications choose them to group items read together, which makes them a useful dimension worth promoting to a real column.

One modelling note on single-table designs, which are common in DynamoDB. A single table holding several entity types will land as one stream containing a mixture, so splitting it by entity type in your silver layer is usually the first transformation worth writing.

Frequently asked questions

Does Airbyte use DynamoDB Streams for change data capture?

No. The source is cursor-based. Deletions are not captured and updates are only seen if they advance the cursor attribute.

Why is an attribute missing from my table?

Discovery did not see it in the sampled items. Refresh the source schema, and check the raw record column, since the value may be present even though no typed column was created for it.

Can I use a nested field as the cursor?

No. Cursor fields must be top level, string or integer, in ISO 8601 or epoch format, and never blank. Single-table designs that nest timestamps need a restructure or a full refresh.

How do I avoid storing AWS keys in Airbyte?

Omit both key fields and attach an IAM role to the workload running the connector. It then falls back to the default credentials chain.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. Parsing the raw records into a modelled table is Spark or SQL work, and it is where the actual modelling happens.

Get your DynamoDB data into Databricks

Check the discovered schema against what your developers say the items contain, pick a cursor that satisfies the rules rather than the one you wish worked, decide what a deletion should mean, and treat the landed table as a bronze layer with the raw record kept for reprocessing.

Airbyte's connector catalog includes 600+ pre-built connectors, so the same source can feed a relational destination when the volume suits it. For that pairing, see DynamoDB to PostgreSQL, and for another schemaless source landing in the same lakehouse, MongoDB to Databricks.

Start syncing now →