Audit Connections with the new Timeline Feature

The new Timeline feature lets you audit Airbyte connections, giving a clear chronological view of sync events, changes, and connection activity.

ENGINEERING

October 24, 2024

8 min read

Natalie Kwong

Summarize with AI:

Audit Connections with the new Timeline Feature

We’re thrilled to announce the launch of a new feature for all our users, the Connection Timeline.

‍

What’s New?

In the past, we focused primarily on the history of sync events (limited to sync, refresh and clear events), which allowed you to track the synchronization of data. While this was helpful, many of you expressed a need for more comprehensive visibility into your connection’s history—beyond just syncs. You wanted a way to track schema changes, connection updates, and other significant events that impact your data flows. And also know the user who applied the changes.

‍

We have now rolled out an early release of the Connection Timeline feature, which is designed to give you a fuller picture of your data connections. The Timeline has been released to all our Airbyte products of Airbyte v1.1.0.

Replacing the "Job History" Tab

The Connection Timeline replaces the "Job History" tab in the UI. It will still be accessible, now as part of the more detailed Timeline page. This new layout is designed to give you easier access to a broader range of events that affect your data connections.

‍

What to Expect

As of now, the Timeline includes events like syncs, refreshes, and clears. You can also see details on which user manually initiated these actions. This gives you more transparency over what’s happening. In addition to those events, we’ve also rolled out:

‍

Settings Changes: Monitor updates made to the connection settings, like schedule frequency
Schema Update: See modifications to schema, like additions of fields or removal of streams

‍

In the future, we’ll be adding additional events that give you further visibility.

Version Changes: Track when and how your connector versions have been updated
Connector Changes: Understand when connector configurations are modified

‍

These updates will provide a more complete history of your data connections, helping you troubleshoot issues faster and understand the evolution of your data flow more clearly.

A Sneak Peek

Here’s a quick look at the Timeline in action.

‍

Sync Events

‍

Sync status
Data and Records moved
Time for sync to complete
For manually triggered syncs, the user who triggered the sync is noted in a “Sync started” event.
Refresh and Clear events follow a similar data structure.

‍

Settings Changes Events

The name of the setting. If multiple settings are changed, the changes are grouped into the same event.
Previous and new values of the connection setting
The user who changed the setting

Connection Enabled/Disabled Events

The reason the connection was disabled (if Airbyte automatically disabled it)
The user who enabled or disabled the connection

Schema Update Events

The reason the connection was disabled (if Airbyte automatically disabled it)
Who enabled or disabled the connection

Migration Steps

For Self-Managed users, we recommend using our Migration Guide to migrate historical jobs to the new Timeline events.

‍

Summary

We believe that the Connection Timeline will significantly improve how you manage and monitor your data connections, making your experience with Airbyte even better.

For more detailed information, feel free to explore our documentation. We’d also love to hear your thoughts on this new feature. If you have any feedback or questions, make sure you join the conversation on the slack community.

About the Author

Natalie Kwong

Natalie Kwong is a Senior Product Manager at Airbyte.