Parquet File to DynamoDB: How to Load Your Data

Loading Parquet files into DynamoDB moves data from a columnar analytical format into a key-value operational store, which is a legitimate thing to want. Your pipeline computes something in a lake, and an application needs to read the result per request at single-digit millisecond latency. That is what DynamoDB is for.

Before you build it, though, there is one thing about the destination that should shape the decision, and it is better discovered here than after the first sync. Airbyte writes each record as a JSON object with four fields, one of which holds your entire row as a blob. You do not get your columns as DynamoDB attributes, which changes what you can query.

Parquet File to DynamoDB at a glance:

Why move data from Parquet files to DynamoDB?

One situation accounts for nearly all of these pipelines, and it is worth being precise about it.

You have a batch process that produces a result per entity: a score per customer, a set of recommendations per user, a configuration per device. It lands as Parquet in object storage because that is what your compute writes. An application then needs to look that result up by key, thousands of times a second, and querying a data lake for that is not viable.

That access pattern, fetch one item by its key, is exactly what DynamoDB is good at and it is also the only pattern this pipeline serves well. If you need to query by anything other than the key, read the section on item structure below before committing, because the answer will affect your design.

What do you need before you start?

Two of these are credentials and two will decide whether the first load completes in minutes or hours:

The right file-based source connector. There is no connector called Parquet. Parquet is a file type you select inside a file-based source, so pick the connector matching where your files live, an S3 bucket, an SFTP server, Azure Blob Storage, then choose Parquet as the format.

An Airbyte-specific AWS user. With read and write permissions on the target tables. Airbyte recommends creating a dedicated user rather than reusing an existing one, which makes permissioning and revocation straightforward.

Provisioned capacity above the default. The connector uses 10 capacity units for both read and write by default. Ten write units is a few hundred small items per minute, which is fine for a trickle and hopeless for a Parquet file holding a million rows. Provision more in the DynamoDB console before the first load, and consider on-demand capacity if the pattern is bursty.

A table name prefix and a naming convention. Each stream is written to its own table, named from a prefix you configure, with the namespace forming part of the name. Decide that convention before the first sync, because renaming tables afterwards means repointing whatever reads them.

Make sure your DynamoDB tables are reachable from wherever Airbyte runs, particularly if they sit in a separate VPC. The check connection tool in the interface is the quickest way to confirm that before you configure anything else. Full field details are in the DynamoDB destination documentation.

How do you build a Parquet to DynamoDB pipeline in Airbyte?

Step 1: Provision capacity and prepare the files

Raise the write capacity on your target tables well above the default before you begin, or switch them to on-demand. Then check the shape of your Parquet files: if they contain nested structs and repeated fields, flatten them upstream, because everything here works better when a row is a simple set of scalar values.

Step 2: Configure the file-based source

Click Sources in the left navigation, then New Source, and select the connector matching your storage, following adding a source. Supply the credentials and path, and choose Parquet as the file format. Whether you can sync incrementally depends on the source connector rather than on Parquet: some track file creation or modification time, others re-read the directory each run, so confirm which behaviour you have before pointing it at a bucket that grows daily.

Step 3: Configure the DynamoDB destination

Click Destinations, then New Destination, and select DynamoDB, following adding a destination. Supply the access key ID, secret access key, region and table name prefix. Leave the endpoint field empty for AWS DynamoDB, or fill it in if you are pointing at a custom endpoint such as a local instance for testing.

Step 4: Choose your sync mode carefully

Click Connections, then New connection, and select your streams and a sync mode. Overwrite mode deletes all previously synced data in the configured table, which is exactly right when each batch supersedes the last and exactly wrong if an application is reading that table while the sync runs. Append avoids the deletion but leaves you managing versions yourself.

If a live application depends on the table, the safer pattern is to load into a new table and switch what your application reads once the sync completes, rather than emptying a table something is actively querying.

What does an item actually look like in DynamoDB?

This is the section to read before deciding anything else. Each stream becomes a table of JSON objects containing four fields: a UUID Airbyte assigns to the record, the timestamp it was pulled from the source, a timestamp for when the sync ran, and a field holding your extracted data as a JSON blob.

Your Parquet columns are inside that blob. They are not DynamoDB attributes. That distinction matters enormously, because DynamoDB's query capabilities work on attributes: you cannot build a global secondary index on a field that lives inside a JSON string, and you cannot filter efficiently on it either. What you can do is read items and parse the blob in your application.

So be honest about your access pattern. If the application fetches by a key and then uses the whole record, this is fine and the blob is irrelevant to it. If the application needs to query by a secondary attribute, this connector will not give you the table design to support that, and you would be looking at writing to DynamoDB directly from your batch job instead. Better to establish that now than after provisioning capacity.

Why is my first load taking so long?

Almost certainly the default capacity. The connector provisions 10 capacity units for both read and write, which is a sensible conservative default and a serious constraint for a bulk load. DynamoDB throttles writes beyond your provisioned capacity, so the sync does not fail, it simply crawls.

Parquet makes this easy to underestimate, because Parquet compresses well and a file that looks modest in object storage can hold a great many rows. Check row counts rather than file sizes when you are estimating how much capacity a load will need.

Raise provisioned capacity in the DynamoDB console before a large sync and lower it afterwards, or use on-demand capacity so the table scales with the burst. Either is cheaper than a sync that runs for six hours against a throttled table, and considerably cheaper than discovering the problem during a launch.

Frequently asked questions

Can I build a secondary index on my data in DynamoDB?

Not on fields from your source data, because they arrive inside a JSON blob rather than as DynamoDB attributes. If you need to query by a secondary attribute, this connector will not produce the table design you need.

Why is my sync so slow?

The connector uses 10 capacity units for read and write by default and DynamoDB throttles beyond provisioned capacity. Raise it in the console before a bulk load, or switch the table to on-demand.

Which connector reads Parquet files?

Whichever file-based source matches where your files live, with Parquet chosen as the file type. There is no standalone Parquet connector.

Is overwrite mode safe on a live table?

No. It deletes all previously synced data in the configured table. If an application reads that table continuously, load into a new table and switch what the application points at once the sync completes.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. Your application will need to parse the JSON blob it reads back, which is a few lines wherever that code already lives.

Get your Parquet data into DynamoDB

Confirm your access pattern is a key lookup, because the JSON blob structure rules out querying by secondary attributes. Then raise the provisioned capacity before the first load, choose your sync mode with the reading application in mind, and settle the table naming convention up front.

Airbyte's connector catalog includes 600+ pre-built connectors, so the same Parquet files can feed an analytical destination alongside this one. For the same file format going somewhere with a very different type model, see Parquet to Elasticsearch, and for DynamoDB as a source rather than a destination, DynamoDB to Databricks.

Start syncing now →