Article

The Drip | November 2022 Airbyte Product Updates

•

December 5, 2022

•

Summarize with ChatGPT

Hey everyone, welcome to the November edition of the Drip where we take you downstream to cover highlights of our change-log, community and anything Airbyte related.

Concluding Hacktoberfest

In November, we concluded our Hacktoberfest contest, which you’ve heard plenty about. We were able to get out 103 new connectors, 75 of which were built using our low-code CDK! Thank you to everyone who contributed and added to our ever growing list of connectors. We also concluded our inaugural State of Data Engineering survey, which we’re pleased to report is the largest survey of data engineers ever (our tease of results)!

Airbyte Core Upgrades

Apart from all the new connectors, Airbyte OSS benefited greatly, with visible improvements to Airbyte Core (Progress Bars!), as well as internal ones (Airbyte Protocol v1)!

On the Airbyte Cloud side, we finally launched Airbyte Cloud in Europe, with a hybrid model that works great for Data Protection, and also launched our Cloud API in Alpha!

Last but not least, get your virtual ticket for our move(data) conference (THIS WEEK!) - speakers, share on Twitter! Awesome raffle prizes to be won!

⌛ Sync Progress Bars!

Davin Chia, Jimmy Ma and Evan Tahler won our most recent Airbyte Hack Day with this one feature - giving you a progress bar for your sync:

‍

🚨 MVP release of the connector builder

OSS Users running Airbyte 0.40.19 or higher can now visit localhost:8000/connector-builder and get to building them low-code, low-frills connectors!

https://www.loom.com/share/acf899938ef74dec8dd61ba012bc872f

This is the first release on our Connector Builder UI roadmap, and is focused on creating a fast feedback loop for connector developers writing YAML connectors. Look out for a full talk on this during our move(data) conference this week! We’ll be iterating on this MVP with feedback from people in our private early access program (closed for now). Meanwhile, find the docs for getting started here!

Wrap Up

And that’s all we have for November’s edition of The Drip. Thanks for reading through. If you have any questions:

Please join our Slack community to talk to us on the Airbyte team as well as other fantastic folks in the community!
Also sign up for our Newsletter to keep up with the state of the art in Data Integration and the broader Data Engineering Ecosystem!

‍

Here are the patch notes from v0.40.18 to v0.40.23:

✨ New and improved features

Headline: MVP release of the connector builder (see above)
New Sources and Promotions
Source Tempo: migrate to lowcode + certify to beta (#18936)
Source Prestashop: Migrate connector from Alpha (Python) to Beta (YAML) (#18927)
New Features for Existing Connectors
Destination Meilisearch: migrating to python and fix integration (#18036)
New Features in Airbyte Core
Add airbyte-cron Helm chart (#18542)
Progress Bar Estimate (#19814)
New Connectors (mostly from Hacktoberfest)
Destination: Apache Doris (#17884)
Destination: Apache Iceberg (#18836)
Destination: Heap Analytics [python cdk] (#18530)
Destination: Redpanda (#18884)
Destination: S3-Glue (#18695)
Destination: Typesense (#18349)
Destination: Yugabytedb (#18039)
Source: ActiveCampaign (#18335)
Source: Aha (#18893)
Source: Ashby (#18334)
Source: Auth0 [python cdk] (#18338)
Source: CallRail (#19407)
Source: ClickUp (#17770)
Source: Clockify [python cdk] (#17767)
Source: CoinGecko Coins (#18248)
Source: Coda [python cdk] (#18675)
Source: Coinmarketcap [low-code CDK] (#18565)
Source: Coin API (#18302)
Source: ConvertKit (#18455)
Source: Copper [python cdk] (#18848)
Source: DataScope [low-code cdk] (#18725)
Source: Dynamodb (#18750)
Source: getLago [low-code cdk] (#18727)
Source: Genesys [python cdk] (#17559)
Source: Google Webfonts [low-code cdk] (#18496)
Source: Intruder [low-code cdk] (#18668)
Source: Mailjet SMS (#18345)
Source: Mailjet Mail API (#18332)
Source: MailerLite (#18336)
Source: Mailersend [low-code CDK] (#18669)
Source: Microsoft Dataverse [python cdk] (#18646)
Source: Nasa APOD [python cdk] (#18394)
Source: News API (#18301)
Source: Lokalise [low-code SDK] (#18522)
Source: Oura (#18224)
Source: Omnisend (#18577)
Source: Pexels API [low-code cdk] (#18854)
Source: Plausible [low-code cdk] (#18657)
Source: Polygon (#19504)
Source: Public APIs [python CDK] (#18471)
Source: Punk API [low-code cdk] (#18724)
Source: Qonto [python cdk] (#17452)
Source: Recreation.gov (#18881)
Source: Rocket.chat [low-code cdk] (#18635)
Source: RD Station Marketing [python cdk] (#18348)
Source: RSS [Python CDK] (#18838)
Source: SAP Fieldglass [low-code cdk] (#18656)
Source: SmartEngage [low-code cdk] (#18701)
Source: Sonar Cloud (#18475)
Source: SFTP Bulk [python cdk] (#17691)
Source: SurveyCTO (#19371)
Source: TMDb [low-code cdk] (#18561)
Source: TVMaze Schedule (#18333)
Source: Twitter (#18863)
Source: Tyntec SMS [low-code cdk] (#18883)
Source: Waiteraid [low-code cdk] (#18165)
Source: Workable (low-code SDK) (#18033)
Source: Workramp (#18843)
Source: Vantage [low-code cdk] (#18665)
Source: Vitally (#18545)
Source: Xero [python cdk] (#18666)
Source: Zendesk Sell [python cdk] (#17888)

🚨 Breaking changes

Airbyte Protocol v1 (#19846): Define a set of well-known data types (#17486)
This adds a set of data type schemas. These will be used in place of the existing JsonSchema primitive types. They provide out-of-the-box validation, and enforce specific formatting on the more interesting data types (e.g. timestamps). Non-primitive types (object, array, oneOf, etc) will still use raw JsonSchema types.
These well-known types map mostly 1:1 with the existing Airbyte data types. Notable differences:
BinaryData is the only new type; it's intended to be used in places where we currently produce a base64 string (e.g. source-postgres' handling of BINARY columns).
Time with/without timezone have been de facto in-use for some time, so we'll make them official
The big_integer and big_number types are being retired, because they weren't used. In the future, we'll add support for large numeric values via airbyte_properties, described below.
Note: Connectors in your instance can't be upgraded via the UI in v 0.40.19-21. Use 0.40.22.

🐛 Bug fixes

Destination Bigquery: updated Check method to test for non-billable project (#19489)
Destination Google Sheets: Fix empty headers list (#18729)
Destination Elasticsearch: enforce ssl connection on cloud (#18341)
Destination Redshift: handle empty s3 bucket_path (#18434)
Source Airtable: improve discovery logic (#18491)
Source Asana: migrate to new SAT, added base HTTP errors handling (#19561)
Source Amplitude: fix empty series for AverageSessionLength stream (#18684)
Source Exchange Rates: Fix handling error during check connection (#18726)
Source Facebook Marketing: reduce request limit after specific error (#18734)
Source Facebook Marketing: increase timeout (#18536)
Source: Facebook Marketing - New default for action_breakdowns, improve "check" speed (#19803)
Source Freshdesk: do not call response.json() inside should_retry (#19349)
Source freshdesk - fix schema (#19496)
Source Postgres: Handle Arrays data types (#16990)
Source Prestashop: prevent users from using http url (#18599)
Source Salesforce: Use Apparent Encoding (#19294)
Source Salesforce: add get_error_display_message for ConnectionError (#18753)
Source Salesforce: add more unsupported bulk entities, fix fallback to rest (#19286)
Source Salesforce: make AccountHistory support BULK API (#19869)
Source Sendgrid: fixed expected records (#19819)
Source SurveyMonkey: added OAuth for Eu and Ca (#17964)
Source Tiktok Marketing: fix test timeout (#18433)
Source Twilio: fix generating date ranges (#19479)
Source Twilio: fixed expected records (#19828)