Connectors

/

AWS CloudTrail

AWS CloudTrail

Engineering Analytics

Sync AWS CloudTrail data anywhere.

AWS CloudTrail is a web service developed to simplify and provide assistance with AWS accounts. Enabling compliance, governance, and operational and risk auditing, it allows users to monitor, log, and document AWS account-related activity in an easily searchable format. With its comprehensive account event history function, CloudTrail helps users analyze and troubleshoot security and operational issues, detect unusual account activity, and much more by increasing visibility into customers’ user and resource activity.

  • Standard
  • Alpha
Sync AWS CloudTrail dataRead the docs
  • 50+Destinations
One connector

Everything AWS CloudTrail can do in Airbyte

  • Sync to your warehouse

    Land AWS CloudTrail data in 50+ destinations on a schedule you control.

  • Incremental syncs

    Pull only the records that changed since the last run instead of reloading everything.

  • Cloud or self-hosted

    Run the AWS CloudTrail connector on Cloud, Self-Managed Enterprise.

Sync capabilities

  • Full Refresh SyncSupported
  • Incremental SyncSupported
  • NamespacesNot supported
  • Available onCloud, Self-Managed Enterprise
  • Destinations50+ Airbyte connectors
  • Connector version1.1.0

Set up in 15 steps

  1. First, navigate to the AWS Management Console and log in to your account.
  2. Once logged in, search for the CloudTrail service and select it.
  3. In the CloudTrail dashboard, select the Trails option from the left-hand menu.
  4. Click on the name of the trail you want to use as your source connector.
  5. In the trail details page, scroll down to the section labeled "Management events" and click on the "Edit" button.
  6. In the "Data events" section, click on the "Add data event" button.
  7. Select the type of data event you want to capture and configure the settings as needed.
  8. Once you have configured the data event, click on the "Save" button to save your changes.
  9. Next, navigate to the Airbyte dashboard and select the "Sources" option from the left-hand menu.
  10. Click on the "Create a new source" button and select the AWS CloudTrail connector.
  11. Enter your AWS access key ID and secret access key in the appropriate fields.
  12. Enter the name of the S3 bucket where your CloudTrail logs are stored.
  13. Enter the name of the CloudTrail trail you want to use as your source connector.
  14. Click on the "Test" button to ensure that your credentials are valid and that Airbyte can connect to your CloudTrail logs.
  15. Once the test is successful, click on the "Create" button to create your AWS CloudTrail source connector in Airbyte.
FAQ

Common questions

What is ETL?

ETL, an acronym for Extract, Transform, Load, is a vital data integration process. It involves extracting data from diverse sources, transforming it into a usable format, and loading it into a database, data warehouse or data lake. This process enables meaningful data analysis, enhancing business intelligence.

What data can you extract from AWS CloudTrail?

AWS CloudTrail provides access to a wide range of data related to AWS account activity and resource usage. The following are the categories of data that can be accessed through the API:

1. Event history: This includes information about all the events that have occurred in an AWS account, such as API calls, console sign-ins, and resource changes.
2. Resource activity: This category includes data related to the usage of AWS resources, such as EC2 instances, S3 buckets, and RDS databases.
3. User activity: This category includes data related to user activity in an AWS account, such as user sign-ins, password changes, and access key usage.
4. Security analysis: This category includes data related to security events in an AWS account, such as failed login attempts, unauthorized access attempts, and changes to security groups.
5. Compliance auditing: This category includes data related to compliance auditing in an AWS account, such as changes to IAM policies, CloudTrail configuration changes, and VPC network changes.

Overall, the AWS CloudTrail API provides a comprehensive view of AWS account activity and resource usage, making it a valuable tool for monitoring and managing AWS environments.

How do I transfer data from AWS CloudTrail?

1. First, navigate to the AWS Management Console and log in to your account.
2. Once logged in, search for the CloudTrail service and select it.
3. In the CloudTrail dashboard, select the Trails option from the left-hand menu.
4. Click on the name of the trail you want to use as your source connector.
5. In the trail details page, scroll down to the section labeled "Management events" and click on the "Edit" button.
6. In the "Data events" section, click on the "Add data event" button.
7. Select the type of data event you want to capture and configure the settings as needed.
8. Once you have configured the data event, click on the "Save" button to save your changes.
9. Next, navigate to the Airbyte dashboard and select the "Sources" option from the left-hand menu.
10. Click on the "Create a new source" button and select the AWS CloudTrail connector.
11. Enter your AWS access key ID and secret access key in the appropriate fields.
12. Enter the name of the S3 bucket where your CloudTrail logs are stored.
13. Enter the name of the CloudTrail trail you want to use as your source connector.
14. Click on the "Test" button to ensure that your credentials are valid and that Airbyte can connect to your CloudTrail logs.
15. Once the test is successful, click on the "Create" button to create your AWS CloudTrail source connector in Airbyte.

What are top ETL tools to transfer data from AWS CloudTrail?

The most prominent ETL tools to transfer data to include: Airbyte, Fivetran, StitchData, Matillion, Talend Data Integration. These tools help in extracting data from various sources (APIs, databases, and more), transforming it efficiently, and loading it into and other databases, data warehouses and data lakes, enhancing data management capabilities.

What is ELT?

ELT, standing for Extract, Load, Transform, is a modern take on the traditional ETL data integration process. In ELT, data is first extracted from various sources, loaded directly into a data warehouse, and then transformed. This approach enhances data processing speed, analytical flexibility and autonomy.

Difference between ETL and ELT?

ETL and ELT are critical data integration strategies with key differences. ETL (Extract, Transform, Load) transforms data before loading, ideal for structured data. In contrast, ELT (Extract, Load, Transform) loads data before transformation, perfect for processing large, diverse data sets in modern data warehouses. ELT is becoming the new standard as it offers a lot more flexibility and autonomy to data analysts.

Explore
All connectors

Start moving AWS CloudTrail data today

Free for 14 days on Airbyte Cloud. Set up the AWS CloudTrail connector once and let Airbyte keep it in sync.

Try it freeTalk to sales