Metabase to DynamoDB: How to Move Your Data

Moving Metabase into DynamoDB gives a service somewhere to look up what exists in your reporting estate. Metabase knows which questions, dashboards and collections it holds, and it answers that through an API one call at a time, which is fine for a person and poor for anything that needs the answer repeatedly.

This guide covers the managed path with Airbyte. Two things shape the build: the credentials expire on a schedule that will outlast anybody's memory of setting them up, and records arrive in DynamoDB as a single blob rather than as queryable attributes.

Metabase to DynamoDB at a glance:

Why move data from Metabase to DynamoDB?

Two situations account for most of these pipelines.

The first is serving the catalogue to something that asks constantly. An internal portal listing available reports, a service that checks whether a dashboard still exists before linking to it, a tool that resolves a question identifier to a name: all of these want a fast lookup by key, which is precisely what DynamoDB does well and what a Metabase API call does not.

The second is keeping a record of the estate as it changes, since dashboards accumulate and nobody deletes them. The pairing is wrong if you want to analyse that catalogue rather than look things up in it, because DynamoDB stores these records in a form that resists exactly that. For governance reporting on your reporting, Metabase to BigQuery is the sensible destination.

What do you need before you start?

Four things, and the last one cannot be changed afterwards:

Metabase credentials, and a decision about which kind. A session token, or a username and password. They behave differently in ways that matter operationally, covered below. The Metabase source documentation sets out both.

A reminder in somebody's calendar. A session token lasts around fourteen days and does not renew itself. Unless you self-host and can extend the session duration, this pipeline needs a human touching it roughly fortnightly, which is a maintenance commitment worth acknowledging before you build it.

AWS credentials and a table with provisioned capacity in mind. Default write capacity is low, and a bulk load against it throttles. Throttling does not announce itself as an error; it presents as a sync that is inexplicably slow.

A decided partition and sort key. This is the irreversible one. The access pattern you choose now is the access pattern the table has forever, and changing your mind means creating another table and loading it again.

If your Metabase instance restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Metabase to DynamoDB pipeline in Airbyte?

Step 1: Settle the access pattern, because it is permanent

Write down the questions the consuming service will ask. Looking up a dashboard by its identifier is a different key design from listing everything in a collection, and DynamoDB will serve whichever one you choose and fight you over the other. Since partition and sort keys cannot be altered once the table exists, ten minutes with the consuming team now prevents a rebuild later.

Step 2: Configure the Metabase source

Click Sources in the left navigation, then New Source, and select Metabase, following adding a source. Supply your instance URL and credentials. Expect the catalogue rather than the data behind it: this pipeline carries the definitions of questions and dashboards, not the rows those questions return when somebody runs them.

Step 3: Configure the DynamoDB destination

Click Destinations, then New Destination, and select DynamoDB, following adding a destination. Supply the region, credentials and table configuration. Raise write capacity before the first load if the table is provisioned rather than on-demand, then lower it again once the backfill is done.

Step 4: Create the connection and avoid overwriting in place

Click Connections, then New connection, select your streams and a sync mode. Overwrite is destructive here, so if a service reads this table continuously, load into a new table and repoint the service once the load has finished. A failed overwrite against a live table leaves consumers with nothing rather than with stale data.

A daily sync is plenty. A reporting catalogue changes when somebody builds a dashboard, which is not an event requiring minute-level freshness.

Why does this pipeline keep needing attention?

Because a Metabase session token expires after roughly fourteen days and nothing renews it for you. The pipeline works perfectly for a fortnight, then fails, usually at a point far enough from setup that the cause is not obvious to whoever is investigating.

Supplying a username and password instead avoids the expiry and introduces a different problem: a new session is generated for each authenticated query. That churn can trigger Metabase's security alerting, and a stream of login notifications arriving in an administrator's inbox is the kind of thing that gets a pipeline switched off by somebody who assumes it has been compromised.

Self-hosting resolves this properly, since session duration is configurable and a longer one turns fortnightly maintenance into an annual chore. Otherwise, pick whichever failure suits you, put the rotation in a calendar rather than in somebody's memory, and warn your Metabase administrator in advance if you go the password route so the alerts are expected rather than alarming.

Why can't you query these records the way you expect?

Because each record lands as one JSON blob rather than as a set of DynamoDB attributes. The fields are all there and perfectly readable once an item is retrieved, but they are contents rather than structure, so DynamoDB cannot see them and cannot index them.

The consequence people hit is global secondary indexes. An index needs a real attribute to build on, and a field inside a blob is not one, so a plan involving an index on collection identifier or creation date simply cannot be implemented against this table. Every access route has to go through the partition and sort key you chose, which is why choosing them carefully matters so much here.

Work with it rather than against it. If a consumer needs a second access path, the honest options are encoding what it needs into the sort key, maintaining a second table keyed differently, or accepting that this table serves one pattern and something else serves the other. Discovering this after the table is full and a service is live is a considerably worse place to have the conversation.

Frequently asked questions

Does this give me the data behind my dashboards?

No. The connector carries the catalogue, meaning the definitions of questions, dashboards and collections. The rows a question returns stay in whatever database Metabase queries.

Why did the pipeline stop after two weeks?

The session token expired. They last around fourteen days and need rotating manually unless you self-host and extend the session duration.

Can I build a global secondary index on these records?

Not on the fields inside them, because records arrive as a single JSON blob and an index needs a real attribute. Plan your access around the partition and sort key instead.

Why is the first load so slow?

Probably write throttling, since default capacity is low and bulk loads exceed it. It looks like slowness rather than failure, so raise capacity for the backfill and lower it afterwards.

Can I do this without writing code?

The pipeline, yes. The consuming service is yours, and it will need to parse the JSON blob itself since DynamoDB cannot do that for it.

Get your Metabase data into DynamoDB

Decide the access pattern before the table exists, because the key design is permanent and records arriving as a blob mean no index will rescue a poor choice. Plan for credential maintenance honestly: a session token expires fortnightly, and the password alternative generates enough logins to worry your Metabase administrator. Raise write capacity for the backfill, and never overwrite a table a live service is reading.

Airbyte's connector catalog includes 600+ pre-built connectors, so a reporting catalogue can reach the services that depend on it. For the same source into an application backend, see Metabase to Convex, and for a database source into the same destination, PostgreSQL to DynamoDB.

Start syncing now →