Grafana Cloud k6 to DynamoDB: How to Move Your Data

Moving Grafana Cloud k6 into DynamoDB puts your performance testing catalogue where AWS-native tooling can read it by key. A deployment gate or an internal service checking which tests exist for a project wants a fast lookup rather than a report.

This guide covers the managed path with Airbyte, and two things are worth saying plainly. The connector carries the catalogue rather than any test results, and this is a small dataset in a destination built for very large ones.

Grafana Cloud k6 to DynamoDB at a glance:

Why move data from Grafana Cloud k6 to DynamoDB?

One situation suits this, and it is narrow enough to check before building.

The good case is serving software in an AWS estate. A pipeline gate asking whether a service has performance tests, or a developer portal showing what exists for a project, wants a key lookup from a store its other components already use.

The poor case is anything analytical, because three small streams in a key-value store answer no questions about trends. If you want to govern the testing estate or see how it changed, Grafana Cloud k6 to Snowflake does that far better.

What do you need before you start?

Four things, and the first two are worth settling before any configuration:

Agreement that results are out of scope. Three streams are available and none carries metrics or run outcomes. The Grafana Cloud k6 source documentation lists them, and it is worth showing to whoever asked.

A named consumer that reads by key. If nothing queries this by identifier at speed, a key-value store is the wrong shape and almost any other destination will serve you better.

A stack token rather than a personal one. Both authenticate, and a personal token is tied to an individual, so it leaves with them and takes your pipeline with it.

A partition key chosen from your lookups. DynamoDB rewards knowing the key you will query by, and a table designed from the source's shape rather than the reader's needs is expensive to use.

If your network restricts traffic by IP, add the Airbyte Cloud IP addresses to the relevant allow list before you begin.

How do you build a Grafana Cloud k6 to DynamoDB pipeline in Airbyte?

Step 1: Name the thing that will read this

Identify the service or tool that will query this table and the key it will use, because that answer justifies the pairing or ends it. A deployment gate looking up tests by project identifier is a real reason. Wanting the data in AWS because everything else is in AWS is a preference, and a preference does not need a pipeline plus a table somebody maintains.

Step 2: Configure the Grafana Cloud k6 source

Click Sources in the left navigation, then New Source, and select Grafana Cloud k6, following adding a source. Supply the API token, which is essentially the whole configuration. Three streams appear, describing your organisations, projects and the tests defined within them.

Step 3: Configure the DynamoDB destination

Click Destinations, then New Destination, and select DynamoDB, following adding a destination. Supply the region, credentials and table naming details. On-demand capacity suits a dataset this small far better than provisioned throughput anybody has to size.

Step 4: Create the connection and protect the readers

Click Connections, then New connection, select your streams and a sync mode. Only full refresh is available, which matters more here than in a warehouse, for the reason below. Daily is generous for a catalogue that changes when somebody creates a test.

Then decide how a failed load should behave, since something is reading this table while the pipeline runs.

Is a key-value store right for three small streams?

Only when something reads by key, which is a narrower condition than it sounds. DynamoDB is excellent at returning a known item quickly at any scale, and a k6 catalogue is a few hundred records describing organisations, projects and tests. The scale argument does not apply, so the access pattern has to carry the decision alone.

When a service genuinely needs that lookup, this is a reasonable arrangement and cheap to run. A gate in a deployment pipeline asking whether a project has tests, called on every build, is exactly the pattern, and running it against a key-value store in the same account is simpler than calling k6 each time.

What it will not do is answer questions. Counting tests per team, finding what was deleted or comparing the estate to last quarter are scans rather than lookups, and a key-value store is a poor place for them. Be honest about which you need, because the same three streams in a warehouse answer the second kind easily and this destination never will.

What does full refresh mean for a table something reads?

A window you should not ignore. Every sync replaces the contents rather than updating them, which is harmless in a warehouse nobody queries at three in the morning and rather less so when a deployment gate reads this table continuously. A load that fails partway leaves a table that is present and incomplete.

The dangerous version is a service that treats a missing record as a negative answer. A gate asking whether a project has performance tests, receiving nothing because the load was halfway through, concludes that it does not and blocks or allows a deployment on the strength of an artefact of timing.

So load into a new table and repoint your reader once the load has finished, rather than refreshing in place under something live. Choose a partition key matching the lookup your service makes, usually the project or test identifier, and have the reader treat an unexpected absence as unknown rather than as no. The data is small enough that all of this is cheap, and the failure it prevents is not.

Frequently asked questions

Can I get test results or metrics?

No. The three streams describe organisations, projects and tests, and none carries run outcomes, metrics or percentiles.

Is DynamoDB overkill for this?

For analysis, yes. For a service doing key lookups inside an AWS estate it is a sensible fit, and the dataset being small costs you nothing.

What happens if a sync fails halfway?

You can be left with an incomplete table, since every sync is a full refresh. Load into a new table and repoint your reader when it completes.

Which token should I use?

A stack token, since a personal one is tied to an individual and stops working when they move on, in a way the failure does not explain.

Can I do this without writing code?

The pipeline, yes. The repoint procedure and the service reading the table are yours, and they are what makes this safe to depend on.

Get your Grafana Cloud k6 data into DynamoDB

Name the service that will read this by key, because that justifies the pairing and nothing else does. Establish that results are out of scope, and use a stack token so the pipeline survives staff changes. Then load into a new table and repoint rather than refreshing under a live reader, choose a partition key matching the lookup, and make sure an absent record is treated as unknown rather than as no.

Airbyte's connector catalog includes 600+ pre-built connectors, so engineering metadata can reach whatever needs to look it up. For the same source into a search engine, see Grafana Cloud k6 to Elasticsearch, and for a key-value source into the same destination, Azure Table Storage to DynamoDB.

Start syncing now →