Article

The Drip | December 2022 Airbyte Product Updates

•

January 6, 2023

•

Summarize with ChatGPT

Hey everyone, welcome to the December edition of the Drip where we take you downstream to cover highlights of our change-log, community and anything Airbyte related.

move(data) Conference Videos are Live!

In December, we organized our first ever move(data) conference, with over 3,300+ attendees from 72 countries and 40 incredible lightning talks from speakers all across the data stack! If you missed it, you can watch Michel's keynote and catch up on the talk videos here.

Airbyte Cloud Gets Custom Connectors!

We're pleased to announce that our Airbyte Cloud platform now supports custom connectors! This means that you can now build and use your own connectors to bring in data from sources that aren't currently supported by Airbyte. This opens up a whole new world of possibilities for data integration and exploration.

This feature is still in alpha stages but if you are interested in signing up for access, please talk to Andy Jih either through our Community Slack!

Controlling Airbyte Cloud through an API

You can now use API calls to fully control Airbyte Cloud and all of its resources. Want to create a new connector? There's an API call for that. Need to update a integration? There's an API call for that too. The possibilities are endless!

But that's not all - this new feature also opens up a world of automation and integration with other systems. Imagine being able to set up a seamless data pipeline without lifting a finger. It's all possible with the Airbyte Cloud API.

Ready to start using the API and taking your data integration to new heights? Head over to our API documentation page to learn more and if you want access, please reach out to Riley Brook through our Community Slack!

Airbyte OSS Gets More Visibility 👀

There is also a new update to our sync UI: the ability to display throughput and progress bars! This means that you can now see exactly how much data has been transferred during a sync, as well as get a visual representation of the sync progress.

This update is designed to make it easier for you to track your sync status and understand how your data is being transferred. Plus, the progress bar is just plain cool to look at!

To see the throughput and progress bars in action, simply start a sync and watch as the data starts flowing. You'll see real-time updates on the amount of data transferred and the sync's progress.

You can see this in action now through Airbyte OSS!

Wrap Up

And that’s all we have for December’s edition of The Drip. Thanks for reading through. If you have any questions:

Please join our Slack community to talk to us on the Airbyte team as well as other fantastic folks in the community!
Also sign up for our Newsletter to keep up with the state of the art in Data Integration and the broader Data Engineering Ecosystem!

Changelog:

🎉 New Sources and Promotions

New Source: Babelforce (#12700)
New Source: Braze [low-code CDK] (#20520)
New Source: Google PageSpeed Insights [low-code CDK] (#19813)
Source Close.com: promoted to Beta (low code) (#18968)
Source Chartmogul: promoted to Beta (low code) (#18978)
Source Delighted: promoted to Beta (low code) (#18988)
Source Facebook Pages: promoted to Beta (low code) (#19788)
Source Gitlab: promoted to Beta (low code) (#20511)
Source Jira: promoted to Beta (low code) (#20128)
Source Metabase: promoted to Beta (low code) (#19236)
Source Monday: promoted to Beta (low code) (#19586)
Source Posthog: promoted to Beta (low code) (#18993)
Source Square: promoted to Beta (low code) (#19369)
Source WooCommerce: promoted to Beta (low code) (#19903)
Source Zenloop: promoted to Beta (low code) (#19624)
New Features for Existing Connectors
Source Amplitude: enable data region selection (#19727)
Source BigCommerce: add order_products stream (#20090)
Source Bing Ads: add fields to schema (#20005)
Source Firebolt: allow reading from views (#16583)
Source Retently: Add campaigns, feedback, outbox, templates streams (#19456)
Source Retently: Add Feedback, NPS Score, Campaigns streams (#19381)
Source RKI-Covid - add history data, incidence rate, cases, death, hospitalization streams (#17607)
Destination Clickhouse: Publish strict encrypt variant (#20171)
New Features in Airbyte Core
UI: Show Source-defined cursorand primary key fields on new Stream Details panel (#20366)
UI: Prevent users from editing existing connection if they have a schema change (#20276)
UI: check that stream is defined when matching (#20443)
UI: New stream details panel (#19219)
UI: Auto-detect schema changes UX flow (#19226)
UI: Add auto-detect schema changes option to connection UIs (#19734)
UI: Display Throughput and Progress Bar in Sync UI (#19193)
UI: new streams table edit modal namespace and stream (#19713)
Low-Code CDK: added SessionTokenAuthenticator (#19716)
Helm Charts: allow for adding extra labels to pods (#19474)
New implementation for AWS Secret Manager for issue (#19690)
Ongoing work on our new Low Code CDK
[Connector Builder] Configuration UI MVP (#20008)
[Connector Builder] Add request options component (#20497)
[Connector Builder] correctly ignore individually touched fields in useBuilderErrors (#20463)
[Connector Builder] Update view styles and add stream delete button (#20464)
[Connector Builder] upgrade orval and use connector manifest schema directly (#20166)

🚨 Breaking changes

Nothing major to worry about for most people

Destination Definitions: move normalization fields to normalizationConfig (#20573)
API changes to support configuring column selection (#20259)

🐛 Bug fixes

Destination S3-Glue: fix parsing empty object in schema (#19907)
Destination ClickHouse: Update Normalization Docker File (#19573)
Source Amplitude: remove data_region as required parameter (#20312)
Source ElasticSearch: avoid too_long_frame_exception (#18134)
Source Facebook Marketing: remove unsupported fields (#20402)
Source Facebook Marketing: fix fields permission error (#20165)
Source Facebook Marketing: update next cursor value at read start (#20131)
Source: Facebook Marketing - action_breakdowns can be empty list - bugfix for #20016 (#20043)
Source Greenhouse: fix SAT from failed expected_records (#20515)
Source Github: move known error handling to GithubAvailabilityStrategy (#19978)
Source Gitlab: revert availabilitystrategy and bump to latest version of the cdk (#20542)
Source Gitlab: fix duplicate records issue (#20023)
Source Gitlab: fix stream schema (#19986)
Source iterable: fix pendulum.parse memory leak (#19913)
Source Kafka: fix missing data (#19587)
Source MySQL: mark unknown column exception as config error (#20289)
Source MySQL: Check for REQUIRE CLIENT privilege in CDC (#20000)
Source Pardot: failed while syncing Visitors stream (#19618)
Source Pinterest: added TypeTransformer to keep the schemas safe (#20242)
Source Postgres: just throw a warning if a cursor column contains null value (#20192)
Source Postgres: handle terminate connection exception with test. (#19887)
Source Postgres: Fix not being able to configure wal2json plugin (unpublished) (#19985)
Source RD Station Marketing: Fix stream analytics conversions (#18826)
Source S3: Check config settings for CSV file format (#20262)
Source Slack - remove OAuth2.0 refresh_token (#19970)
Source-Snapchat: always set the full state object (#20498)
Source Twilio: fix failed SAT and exposed auth_token (#20430)
Source Zendesk Support: handle 429 response behavior when retrying (#19967)
Source Zoom: upgrade CDK version (#19939)