AWS CloudTrail to Elasticsearch: How to Move Your Data
Index AWS CloudTrail into Elasticsearch with Airbyte. Plan availability limits, mapping deeply nested events, and the 90 day lookup window.
Indexing AWS CloudTrail into Elasticsearch is what you do when the question is not how many but who, and when, and what else did they touch. Security investigations are free-text searches across account activity, and the CloudTrail console is built for filtering rather than for the kind of exploratory search an incident actually needs.
Before designing anything, two constraints decide whether this pipeline suits you. The Elasticsearch destination runs on Airbyte Core and PyAirbyte and is not offered on the Standard, Plus, Pro or Enterprise Flex plans. And the CloudTrail lookup API returns roughly the last 90 days, so your index is the archive rather than a mirror.
AWS CloudTrail to Elasticsearch at a glance:
Why move data from AWS CloudTrail to Elasticsearch?
Two situations account for most of these pipelines, and both are about investigation rather than reporting.
The first is incident response. Somebody asks what a particular principal did last Tuesday, or which calls touched a resource whose name you half remember. Those are free-text searches across nested event structures, with relevance and partial matching, which is what a search engine does and a filtered console view does not.
The second is retention for audit. CloudTrail's lookup reaches back about 90 days, so anything a compliance question asks about last year is not available from that interface. An index that keeps accumulating answers it. If you want counts and trends rather than search, an analytical database is a better destination.
What do you need before you start?
The first item can rule the pipeline out entirely, so confirm it first:
A self-managed Airbyte deployment. The Elasticsearch destination is available on Airbyte Core and PyAirbyte, and not on the paid Cloud tiers. No configuration works around this.
AWS credentials with CloudTrail read access. An access key ID and secret access key, plus the region. The connector is built on the Boto3 CloudTrail client, so it behaves like any other AWS SDK caller. The AWS CloudTrail source documentation covers the fields.
Acceptance that you get management events only. Insight events are not supported by this connector. If what you wanted was object-level access logging for an investigation, this pipeline will not provide it and it is better to know now.
An index mapping you have designed. CloudTrail events are deeply nested, carrying identity structures, request parameters and response elements. Deciding which fields are analysed text and which are keyword fields before indexing anything is the difference between a searchable archive and a sprawling mapping you have to rebuild.
Finally, a plan for accounts and regions, because lookup requests are limited to two per second per account per region and most organisations have several of both.
How do you build a CloudTrail to Elasticsearch pipeline in Airbyte?
Step 1: Design the mapping and the index layout
Decide which fields investigators actually search. Event names, principal identifiers and resource names are usually keyword fields for exact matching, while error messages and free-text parameters benefit from analysis. Then decide whether accounts and regions share an index with a field distinguishing them, which makes cross-account search trivial, or get their own, which makes retention and access control simpler. Changing your mind later means reindexing.
Step 2: Configure the CloudTrail source
Click Sources in the left navigation, then New Source, and select AWS CloudTrail, following adding a source. Supply the access key, secret key, region and start date. Remember the start date cannot reach back further than the lookup window allows, so setting it to two years ago does not give you two years of events. Repeat per account and region you are covering.
Step 3: Configure the Elasticsearch destination
Click Destinations, then New Destination, and select Elasticsearch, following adding a destination. Supply your endpoint and authentication. Create the index with the mapping from step 1 first, so the connector writes into a schema you designed rather than one dynamic mapping invented from the first document it saw.
Step 4: Use incremental sync and sync often
Click Connections, then New connection, and choose an incremental sync mode. Frequency here is not about how fresh anyone wants the index, it is about staying ahead of a source window that keeps moving. Daily against a 90 day window leaves plenty of margin; monthly followed by a pause loses events permanently.
The connector retries gracefully when throttled, but if throttling errors continue after multiple retries, for instance because several connector instances share an account and region, the sync fails. Alert on that rather than assuming retries always save you.
Why does the index mapping matter so much here?
Because CloudTrail events are among the most deeply nested documents you are likely to index. An event carries an identity structure describing who made the call, the request parameters, sometimes response elements, plus source address, user agent and resource references. Dynamic mapping will happily guess at all of it and produce a very wide mapping full of fields nobody queries.
The distinction that matters is analysed text versus keyword. An event name or an account identifier should be a keyword field, so that filtering and aggregation work exactly. A free-text error message benefits from analysis. Get these the wrong way round and your filters return nothing while your searches match everything, which on an audit index is worse than useless because it is misleading.
Spending an hour on the mapping before the first sync is the best-value hour in this build, because reindexing an audit archive that has been accumulating for months is a job nobody wants. It is also the step that determines whether an investigator can find what they need under pressure.
Why does the 90 day window change your design?
Because it makes this an archive rather than a replication. Airbyte can only read what the lookup API still returns, so your index starts accumulating from the day of your first sync and nothing recovers what came before it.
That argues strongly for setting this up before you need it. The value of a security archive is entirely in its age, and the difference between configuring it this quarter and next is a quarter of history you will never have when an investigation asks for it.
It also makes a paused connection unusually expensive. On most pipelines a pause produces staleness you catch up on. Here, if the pause outlasts the window, those events are unrecoverable from this source, so a stalled sync is an incident rather than a backlog. Alert on it accordingly.
Frequently asked questions
Why can I not find the Elasticsearch destination in Airbyte Cloud?
It is available on Airbyte Core and PyAirbyte, and not on the Standard, Plus, Pro or Enterprise Flex plans. This pipeline needs a self-managed deployment.
Can Airbyte index CloudTrail events older than 90 days?
No. The lookup API returns roughly the last 90 days, so the index starts at your first sync. An earlier start date does not recover events outside that window.
Should I use one index per account or one shared index?
A shared index with account and region fields makes cross-account investigation simpler. Separate indices make retention and access control simpler. Decide before the first sync, because switching means reindexing.
Why did my sync fail with throttling errors?
Lookup requests are limited to two per second per account per region. The connector retries, but if errors persist, for example because several connector instances share an account and region, the sync fails.
Can I do this without writing code?
The Airbyte setup is entirely UI-driven. You will want to define the index mapping in Elasticsearch, which is short JSON and the most important part of the build.
Get your CloudTrail data into Elasticsearch
Check your Airbyte plan first, because that decides everything else. Then design the mapping before indexing anything, confirm management events are what you need, and set the pipeline up before the compliance requirement arrives rather than after. Sync often enough that the moving window never overtakes you.
Airbyte's connector catalog includes 600+ pre-built connectors, so audit activity can sit beside deployment and incident data. For the same source aggregated rather than searched, see AWS CloudTrail to ClickHouse, and for the same destination fed by a relational source, MySQL to Elasticsearch.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.