Azure Table Storage to Databricks: How to Move Your Data

Moving Azure Table Storage into Databricks is worth setting expectations about, because the shape of what arrives is unusual. Table Storage holds non-relational structured data, and there is no efficient way to read a schema for a given table, so the connector does not try.

What you get instead is a generic schema shared by every stream: one data property holding all the properties of each row. For a destination with weak semi-structured handling that is painful. For a lakehouse it is fine, because parsing that in Spark is ordinary work rather than a workaround.

Azure Table Storage to Databricks at a glance:

Why move data from Azure Table Storage to Databricks?

Two situations account for most of these pipelines.

The first is that Table Storage cannot answer analytical questions about its own contents. It is built for fast key-based access to large volumes of structured rows, not for aggregating across them, so anything shaped like how many or grouped by needs the data somewhere else.

The second is joining and machine learning. Application or telemetry data sitting next to your other sources, in a form Spark and notebooks can work with, under Unity Catalog governance. That combination is what makes a lakehouse the natural destination when the landed shape is semi-structured.

What do you need before you start?

One of these will disappoint a security-minded colleague, so raise it early:

Your storage account name and access key. Found in the Azure portal under your storage account. Airbyte recommends creating a restricted key specifically for its access so you control what it can reach. The Azure Table Storage source documentation covers where to find them.

Acceptance that shared access key authentication is not supported. The connector does not support it yet, so if your organisation's policy is to issue scoped, time-limited shared access signatures rather than account keys, this is a conversation to have before you build rather than after.

A decided list of tables. You choose which tables to replicate, and because every one lands with the same generic shape, selecting deliberately is how you keep the destination comprehensible rather than a pile of identical-looking tables.

Knowledge of what your rows actually contain. Since no schema is read and no type mappings are applied, nothing in the pipeline will tell you. Ask whoever writes to these tables what the properties are and what types they hold, because that conversation becomes your parsing layer.

On the Databricks side each stream becomes a table in Unity Catalog and the connector needs permission to create Volumes for Avro staging. If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list.

How do you build an Azure Table Storage to Databricks pipeline in Airbyte?

Step 1: Document what each table holds

Sit down with whoever owns the application writing to these tables and list the properties each one uses, including optional ones and any that changed type over the years. Nothing in this pipeline discovers that for you, so this list is the specification for your parsing layer and the only documentation the dataset will ever have.

Step 2: Configure the Azure Table Storage source

Click Sources in the left navigation, then New Source, and select Azure Table Storage, following adding a source. Supply the storage account name and the restricted access key you created, then run the connection test.

Step 3: Configure the Databricks destination

Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply your workspace details, the catalog and schema, and authentication. Because the output is a generic shape rather than a modelled schema, the tables Airbyte creates are deliberately simple and all the structure work happens after they land.

Step 4: Select tables and a sync mode

Click Connections, then New connection, select the tables from step 1 and choose a sync mode for each. Both full refresh and incremental are available. Rate limits are unlikely to constrain you here, so schedule against how often the data actually changes rather than around throughput.

Run one table first and look at the result before extending. The generic schema means every table looks alike in the catalogue, so confirming that the data object contains what you expect is worth doing once per table shape rather than assuming.

Why is every stream the same shape?

Because Table Storage does not have a schema to read, and there is no efficient way to derive one. Rather than sampling rows and guessing, the connector uses a data property holding all the properties for any given row, which produces the same generic schema for every stream.

That is an honest design rather than a limitation, and it is worth comparing with the alternative. A connector that sampled rows to infer a schema would miss properties present on only a fraction of them, which is a failure mode this approach avoids entirely: nothing is dropped because nothing is inferred.

The cost lands on you. No explicit data type mappings are applied, so a number written by your application is not guaranteed to arrive as a number your queries can aggregate. Treat the landed table as a bronze layer, parse the data object into typed columns in a silver table, and cast deliberately using the list you wrote in step 1.

How should you model the result?

Anchor on Table Storage's own key structure. Every entity carries a partition key, which groups related entities together, and a row key identifying it within that partition. Together they are the natural unique identifier, and they are what your parsed table should be keyed on.

The partition key is also usually meaningful rather than arbitrary, because applications choose it to group entities that are read together: a customer identifier, a device, a date. That makes it a useful dimension in its own right, and worth promoting to a proper column early rather than leaving inside the data object.

Keep the raw layer underneath. When your application starts writing a new property, you reprocess history from what you already hold rather than starting from the day you noticed. On a source with no schema and no type mapping, that raw table is the only record of what the pipeline actually received.

Frequently asked questions

Why do all my tables have the same schema?

Because Table Storage has no schema to read efficiently, so the connector uses a generic one where a data property holds all properties for each row. Parse it into typed columns downstream.

Can I use a shared access signature instead of the account key?

Not currently. Shared access key authentication is not supported by this connector yet, so plan on a restricted storage account key.

Are data types converted for me?

No. The connector applies no explicit data type mappings, so casting values into the types you want is work for your downstream models.

Can I sync incrementally?

Yes. The source supports both full refresh and incremental syncs, and you choose which tables to replicate.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. Parsing the data object into a typed table is Spark or SQL work, and it is where the actual modelling on this pipeline happens.

Get your Azure Table Storage data into Databricks

Raise the authentication constraint early, document what each table holds because nothing else will, and treat the landed tables as a bronze layer. Then parse the data object into typed columns, promote the partition key to a real column, and keep the raw so history can be reprocessed when the application gains a property.

Airbyte's connector catalog includes 600+ pre-built connectors, so application data can sit alongside your other sources in one lakehouse. For another schemaless source landing the same way, see Firebase Realtime Database to Databricks, and for a key-value store with inferred rather than generic schemas, DynamoDB to Databricks.

Start syncing now →