Azure Table Storage to Kafka: How to Move Your Data

Moving Azure Table Storage onto Kafka publishes rows from a cheap key-value store where other systems can react to them. Table Storage is often where an application quietly accumulated state, and several services wanting that state should not all be querying it directly.

This guide covers the managed path with Airbyte. Two things shape the build: every row arrives with its properties bundled into a single field, and the credential this connector accepts is broader than the one Azure would rather you used.

Azure Table Storage to Kafka at a glance:

Why move data from Azure Table Storage to Kafka?

Two situations account for most of these pipelines.

The first is fan-out from a store that was never meant to serve many readers. Table Storage is inexpensive and simple, and several services polling it independently is the arrangement a bus exists to replace.

The second is moving off Azure gradually, publishing state that other platforms can consume. If a single system needs this and wants lookups by key rather than a stream of changes, Azure Table Storage to DynamoDB is a more direct arrangement.

What do you need before you start?

Four things, and the second deserves a conversation with whoever owns your Azure estate:

Three values from the Azure Portal. The storage account name from the overview tab, an access key from the access keys tab and the endpoint suffix from the endpoint tab. The Azure Table Storage source documentation walks through where each one lives.

Acceptance that the key is account level. Shared access key authentication is not supported by this connector yet, so the restricted credential Azure would recommend is not currently an option here.

Topics created in advance. The destination writes to topics that already exist, and one per table keeps consumers simple rather than forcing them to filter a shared stream.

Consumers prepared for a nested payload. Because of how this source handles schema, the useful content sits one level deeper than most consumers assume.

If your cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build an Azure Table Storage to Kafka pipeline in Airbyte?

Step 1: Decide how to contain an account-level key

Since you cannot narrow the credential itself, narrow what it can reach. Keeping the tables you replicate in a storage account that holds nothing else means the key grants access to exactly the data you intended to share, which is the containment the connector cannot give you. Agree a rotation schedule while you are there, because an account key that nobody ever changes is the version of this that ages badly.

Step 2: Configure the Azure Table Storage source

Click Sources in the left navigation, then New Source, and select Azure Table Storage, following adding a source. Supply the account name, access key and endpoint suffix. The endpoint suffix is the field people skip, and a sovereign or government cloud uses a different one from the commercial default.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and topic configuration, with a topic per table. Messages are JSON wrapping each record with its identifier and stream name.

Step 4: Create the connection and read one message properly

Click Connections, then New connection, select your tables and a sync mode. Then consume a single message and look at its structure before anybody writes a consumer, because the shape is not the one most people expect and five minutes here saves a rewrite.

This connector is community maintained, so run it for a fortnight before consumers depend on it.

What shape do your messages actually have?

Deeper than you would guess, for a reason the documentation explains honestly. Table Storage holds non-relational structured data and there is no efficient way to read a schema for a given table, so rather than guessing, this source uses one generic schema for every stream and puts all of a row's properties inside a single data property.

For Kafka that produces two layers of wrapping. The message carries an envelope with the record, its identifier and the stream it came from, and inside that record the columns you care about sit within the data property rather than at the top level. Neither layer is difficult once you know, and a consumer written against an assumed shape will find nothing.

It also means there is no schema contract to rely on. Rows in the same table can carry different properties, because that is what Table Storage permits, so consumers should read the fields they need and tolerate their absence rather than validating a structure. Publish an example message alongside the topic name, since that is the documentation your consumer teams will actually use.

Why can you not restrict the credential?

Because the connector does not yet support the mechanism Azure provides for it. The documentation recommends creating a restricted key so you can control what Airbyte reaches, and then notes that shared access key authentication is not supported by this connector yet, which leaves an account key as the practical option.

That is a wider grant than a careful team would choose. An account key opens the storage account rather than a table, so the credential sitting in your pipeline configuration can reach more than the data you meant to publish, and no amount of care in the connector changes what the key itself permits.

So contain it elsewhere. Put the tables you replicate in their own storage account, so the key's reach and your intended scope are the same thing, and rotate it on a schedule somebody owns. Check the connector's documentation before your next review too, since the word yet suggests this may change and a narrower credential would be worth adopting when it does.

Frequently asked questions

Why are my columns not at the top level?

The source uses a generic schema and puts all row properties inside a data property, because there is no efficient way to read a schema from Table Storage.

Can I use a shared access signature?

Not at present, since shared access key authentication is not supported by this connector yet. Contain the account key by scoping the storage account instead.

The connection fails and my key is correct.

Check the endpoint suffix, which is a separate field and differs outside the commercial cloud. It is the value people most often skip.

Do all rows in a table have the same fields?

Not necessarily, since Table Storage does not require it. Consumers should read the fields they need and tolerate absence rather than validating a fixed structure.

Can I do this without writing code?

The pipeline, yes. The consumers are yours, and reaching into the data property correctly is the part worth getting right first time.

Get your Azure Table Storage data into Kafka

Contain the account key by keeping replicated tables in their own storage account, since the restricted credential Azure recommends is not supported here yet, and agree a rotation schedule. Set the endpoint suffix deliberately. Then consume one message before anybody writes a consumer, because row properties sit inside a data property rather than at the top level, and publish that example beside the topic name.

Airbyte's connector catalog includes 600+ pre-built connectors, so state in a simple store can reach every system that needs it. For a relational source onto the same bus, see PostgreSQL to Kafka, and for the same source into a key-value store, Azure Table Storage to DynamoDB.

Start syncing now →