DynamoDB to Databricks: How to Move Your Data
Move DynamoDB into Databricks with Airbyte. How schema inference samples your table, strict cursor rules for incremental sync, and IAM role based access.
Syncing DynamoDB to Databricks means turning a schemaless key-value store into typed columns in a lakehouse, and the interesting part happens before any data moves. DynamoDB tables do not have schemas. Databricks tables do. Something has to invent one, and that something is Airbyte's discovery phase.
Understanding how it does that is the difference between a pipeline you trust and one that silently drops fields. Everything else here, the credentials, the sync modes, the Unity Catalog prerequisites, is ordinary setup work. The schema inference is the part worth your attention.
How the schema gets invented
The connector scans your table with a limit of 1,000 items. If the returned data exceeds 1MB, it scans again with the same limit, repeating until it has at least 1,000 items. It then combines every top-level attribute it saw across those items and infers each attribute's type from its value.
Two consequences follow directly. An attribute that appears only on items outside that sample will not be in the catalogue, and therefore will not land in Databricks. On a table where two percent of items carry a refund_reason attribute, a 1,000 item sample may miss it entirely.
And the connector assumes an attribute has the same type everywhere. DynamoDB does not enforce that. If user_id is a number on older items and a string on newer ones, discovery picks whichever it saw and the pipeline behaves unpredictably from then on. Neither of these is a bug. They are the necessary cost of inferring structure from a store that has none. The mitigation is to review the discovered schema against what you know about the table rather than accepting it, and to re-run discovery after any deploy that adds attributes.
Cursor rules are strict
Incremental sync needs a cursor field, and the constraints are tighter than on most sources. It must be a top-level attribute, not nested inside a map. It must be a string or an integer. Only ISO 8601 and epoch formats are recognised as timestamps. And it must never be blank on any item, because a blank cursor leaves comparison behaviour entirely to DynamoDB's own ordering rules, which is not a place you want your pipeline's correctness to live.
If your access pattern buries the timestamp inside a nested attribute, which is common in single-table designs, you cannot use it as a cursor without changing the item structure. Full refresh is the fallback, and on a large table that is expensive in read capacity.
Credentials and landing the data
Supply an access key ID and secret access key for IAM user based access. Omit both and the connector falls back to the default credentials provider, which picks up the role attached to the workload running the container. On self-managed Airbyte in EKS or ECS the second option is better: no long-lived keys in the configuration, and permissions managed as an IAM role like everything else in the account.
On the Databricks side each stream becomes a table in Unity Catalog, and the connector needs permission to create Volumes for Avro staging. Airbyte preserves the raw record alongside the typed columns, which matters here more than usual: when discovery misses an attribute, the raw payload is where you find out. Build your downstream models against the typed columns, but keep the raw column available, because on a schemaless source it is the only audit trail for what the pipeline actually received.
Frequently asked questions
Does Airbyte use DynamoDB Streams for CDC?
No. The DynamoDB source is cursor-based. Deletes are not captured, and updates are only seen if they advance the cursor attribute.
Why is an attribute missing from my Databricks table?
Discovery did not see it in the sampled items. Refresh the source schema, and check whether the attribute is sparse enough that a 1,000 item sample would plausibly miss it.
Can I use a nested field as the cursor?
No. Cursor fields must be top level. Single-table designs that nest timestamps need either a restructure or a full refresh.
How do I avoid storing AWS keys in Airbyte?
Omit both key fields and attach an IAM role to the workload running the connector. The connector then falls back to the default credentials chain.
Get your DynamoDB data into Databricks
The pipeline itself is simple. The schema inference is what deserves your attention, because a store with no schema and a lakehouse that requires one have to meet somewhere, and that somewhere is a 1,000 item sample. Check what discovery found, keep the raw payload, and pick a cursor that satisfies the rules rather than the one you wish worked.
For the same lakehouse fed by a source that does have a schema, PostgreSQL to Databricks is a useful contrast, and ClickHouse to Databricks covers a columnar source landing in the same place.
Integrate with 600+ apps using Airbyte
Move data from 600+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.