AWS CloudTrail to Amazon Redshift: How to Move Your Data

Moving AWS CloudTrail into Amazon Redshift keeps account activity inside AWS while making it queryable for far longer than the console allows. The CloudTrail lookup API returns events from roughly the last 90 days, so any question spanning a year, or proving who did what to an auditor, cannot be answered from event history alone.

This guide covers the managed path with Airbyte. Three constraints decide whether this suits your account: which events you get, how fast you can read them, and the fact that Redshift cannot be loaded without a staging bucket.

AWS CloudTrail to Amazon Redshift at a glance:

Why move data from AWS CloudTrail to Amazon Redshift?

Two situations account for most of these pipelines.

The first is retention beyond the lookup window. Audit and compliance work asks what happened, when, and by whom, over periods far longer than 90 days, and a warehouse is where that record lives once the console can no longer show it to you.

The second is staying inside AWS. If your analytical estate is already Redshift, keeping audit data in the same account and the same security boundary is an easier conversation than exporting it elsewhere, and analysts already have the credentials.

What do you need before you start?

Two of these are expectations rather than configuration, and both can change whether the project is worth doing:

AWS credentials with CloudTrail read access. An access key ID and secret access key, plus the region you are reading. The connector is built on the Boto3 CloudTrail client, so it behaves like any other AWS SDK caller. The AWS CloudTrail source documentation covers the fields.

An S3 staging bucket and IAM permissions. The Redshift destination writes data to S3 as files with a manifest, then issues a COPY command, which is Redshift's own recommended approach. You need put, get, delete and list on the bucket, and it should sit in the same region as the cluster.

Acceptance that you get management events only. Insight events are not supported by this connector. If what you wanted was object-level access logging or function invocation records, this pipeline will not provide them and it is better to know now.

A plan for multiple accounts and regions. Lookup requests are limited to two per second, per account, per region. Most organisations have several of both, so decide early how many connections you will run, because piling several against the same account and region is a documented way to fail syncs.

Note also that an SSH tunnel to Redshift covers the SQL connection only. Staging traffic goes to S3 over HTTPS regardless, so a network design assuming everything passes through a bastion is wrong.

How do you build a CloudTrail to Redshift pipeline in Airbyte?

Step 1: Decide your account and region coverage

List the accounts and regions whose activity you actually need. Because the rate limit is scoped per account and region, coverage determines both how many connections you build and how long each sync takes. An organisation with twelve accounts across three regions is a very different project from a single account in one region.

Step 2: Configure the CloudTrail source

Click Sources in the left navigation, then New Source, and select AWS CloudTrail, following adding a source. Supply the access key, secret key, region and a start date. Bear in mind the start date cannot reach back further than the lookup window allows, so setting it to two years ago does not give you two years of events.

Step 3: Configure the Redshift destination

Click Destinations, then New Destination, and select Redshift, following adding a destination. Supply the cluster host, database, schema and credentials, along with your S3 bucket and IAM key. Staging files are removed after a successful COPY by default, and can be retained if you need them for debugging.

Step 4: Use incremental sync and sync often

Click Connections, then New connection, and choose an incremental sync mode. Frequency here is not about dashboard freshness, it is about staying ahead of a source window that keeps moving. Syncing daily against a 90 day window leaves plenty of margin; syncing monthly and then pausing for a quarter loses events permanently.

The connector retries gracefully when throttled. If throttling errors continue after multiple retries, for instance because several connector instances share an account and region, the sync fails, so alert on that rather than assuming retries always save you.

Why does the 90 day window change your design?

Because it makes this an archive pipeline rather than a replication one. Airbyte can only read what the lookup API still returns, so your Redshift tables start accumulating from the day of your first sync and nothing recovers what came before.

That is a good argument for setting this up before you need it. The value of an audit archive is entirely in its age, and the difference between configuring it this quarter and next is a quarter of history you will never have. If a compliance requirement is coming, the pipeline should predate it.

It also means a paused connection is more expensive than usual. On most pipelines a pause produces staleness you can catch up on. Here, if the pause outlasts the window, those events are unrecoverable from this source, so treat a stalled sync as an incident rather than a backlog.

What does two requests a second actually mean?

It is the ceiling on this pipeline, and it is low. The lookup API returns a limited number of events per request, so two requests per second per account per region puts a hard bound on how many events you can pull regardless of how much compute you throw at it or how large your Redshift cluster is.

On a busy AWS account generating a large volume of management events, that ceiling can be lower than your event rate, which means you never catch up and the moving window eventually overtakes you. Measure your event volume before committing, because this is the failure that is invisible until the gap is permanent.

Where volume is genuinely high, CloudTrail's own delivery to S3 is the mechanism designed for it, and since you already need a staging bucket for Redshift, that route may fit your architecture more neatly than this connector does. Reaching that conclusion before building is much cheaper than after.

Frequently asked questions

Can Airbyte backfill CloudTrail events older than 90 days?

No. The lookup API returns roughly the last 90 days, so your archive starts at your first sync. Setting an earlier start date does not recover events outside that window.

Do I need an S3 bucket for this?

Yes. The Redshift destination stages data in S3 and loads it with COPY, which is Redshift's recommended approach. There is no direct insert path.

Does this include S3 object-level or Lambda invocation events?

No. Only management events are available, and Insight events are not supported by this connector. If you need data events, this is not the right pipeline.

Why did my sync fail with throttling errors?

Lookup requests are limited to two per second per account per region. The connector retries, but if errors persist, for example because several connector instances share an account and region, the sync fails.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. You will want Redshift views over the event records, which is where your audit queries belong.

Get your CloudTrail data into Amazon Redshift

Confirm that management events are what you need, measure your event volume against the two per second ceiling, set up the staging bucket in the right region, and start the pipeline before the compliance requirement arrives. Then sync often enough that the moving window never overtakes you.

Airbyte's connector catalog includes 600+ pre-built connectors, so audit activity can sit beside deployment and incident data. For aggregation in a columnar engine, see AWS CloudTrail to ClickHouse, and for search over the same records, AWS CloudTrail to Elasticsearch.

Start syncing now →