Article

The Drip | July 2022 Airbyte Product Updates

•

August 17, 2022

Hey everyone, welcome to the July edition of the Drip where we take you downstream to cover highlights of our change-log, community and anything Airbyte related.

Airbyte Turns Two

A couple weeks ago, we celebrated Airbyte’s second birthday! In celebration, the team took two days off. We also want to thank the community for being with us these last two years and helping us achieve our vision of solving data integration. Here’s to many more!

Connectors Progress

In July, we had a lot of movement with connectors! GitHub, Google Analytics, Instagram, TikTok Marketing, Bing Ads, and S3 have been moved to General Availability. These are just the highlights but you can take a look at the actual patch notes to see the full list of changes we’ve made!

Octavia CLI Improvements

We’ve made a lot of improvements to the Octavia CLI but here are some notable changes.

You can now set custom HTTP headers on your requests made to the Airbyte server! If you have an instance with basic auth or IAP, this will allow you to now use the CLI which is a significant step to make the CLI compatible with Airbyte Cloud once the public API is released.

Other improvements we’ve made is the ability to enable normalization or the use of custom DBT transformations within your yaml configurations. You also now have the ability to easily switch between different Airbyte instances you may have and deploy the same configuration to multiple instances.

If you haven’t used the CLI just yet, check out the full docs here.

Per-Stream State to OSS

The team has been working hard on getting Per-Stream State into OSS and the time has finally come! Per-Stream state unlocks the ability to sync data only for affected streams, which also allows for a future where incremental sync will just become the norm! We are very excited about this and will continue to expand on. You can check out the progress of this here. Docs are coming soon.

Open Telemetry Metrics

We have added an OpenTelemetry interface to Airbyte OSS so that you can export some built-in metrics when using Airbyte at scale. Some actionable metrics have also been added which you can check out here.

As an example, we can run Airbyte locally along with an Open Telemetry Collector. The Open Telemetry Collector will expose port 4317 to the localhost as the receiving endpoint for metrics to be sent to. You can check out the docs here on how to setup Open Telemetry and start collecting metrics!

Wrap Up

And thats all we have for July’s edition of The Drip. Thank’s for reading through. If you have any questions:

Please join our Slack community to talk to us on the Airbyte team as well as other fantastic folks in the community!
Also sign up for our Newsletter to keep up with the state of the art in Data Integration and the broader Data Engineering Ecosystem!

‍

Full Patch notes:

New and improved features

Source GitHub: now GA

Source Google Analytics: now GA (#13957)

Source Instagram Ads: now GA (#14162)

Source Tiktok Marketing: now GA (#14358)

Source Bing Ads: now GA (#13679)

Source S3: now GA

Source Postgres now in beta (#14326)

Source Recharge now in beta

Source Klaviyo now in beta

Source Notion now in beta

Source Snapchat Marketing now in beta, with stats streams (#14577)

New alpha source: Google Analytics Data API v1 (#12701)

New alpha source: Glassfrog (#13868)

New alpha source: Timely #13292 (#14335)

New alpha source: Kyriba (#12748)

New alpha source: Elasticsearch (#14118)

New alpha source: Orbit.love (#13390)
‍

Octavia CLI improvements (full docs):

Users should be able to easily switch between Airbyte instances and deploy the same configurations on multiple instances.

Users can enable normalization or custom DBT transformation from their yaml configurations.

Users can set custom HTTP headers on the requests made to the Airbyte server. It allows users with instances secured with basic auth or IAP to use the CLI, and it’s a significant step to make the CLI compatible with Airbyte Cloud once we release the public API.

Users can import existing remote resources to a local octavia project with octavia import

Users can get existing resources (#13254)

Users can retrieve the JSON configuration from a remote resource using octavia get , this can be useful for some scripting / orchestration use cases.

Core Airbyte features:

Per Stream State

Per-Stream state new flow (#14634)

Release per stream to the OSS project (#15008)

Display new per-stream and global state to users when viewing connection settings (#15020)

Base Normalization: handle airbyte_type from stream schema in normalization (#13591)

Security: Upgrade platform to openjdk:19-slim-bullseye (#14971)

Self Hosting: Refactor OSS Helm Charts (#14794)

Monitoring: We’ve officially added an Open Telemetry interface to Airbyte OSS, along with a set of actionable metrics to use when running OSS Airbyte at scale!