Article

Everyone has a Postgres connector. So why use Airbyte’s?

Talia Moyal

•

August 11, 2022

•

5 min read

Summarize with ChatGPT

Summarize with Perplexity

Across our 2,000+ open-source daily active users, Postgres has been our most used connector, even in its alpha state. Airbyte moves more than 200TB of Postgres databases monthly, and this grows around 30% each month. As a matter of fact, Postgres was our very first connector on the platform, released in October of 2020!

Today is an important day for all Airbyte users, but also for all Postgres users. We’re excited to announce the release of Postgres to general availability. This means a Postgres connector that works reliably with a 99% SLA, live technical support and pricing that helps you scale.

Let’s unpack what this means and why we think our Postgres connector will become the market’s go-to connector for all things Postgres.

Why Postgres was the most used connector, even in alpha

When we spoke to our users about why they were using our alpha Postgres connector to replicate their production databases we understood that there are no other good offerings in the market. Current solutions are prohibitively expensive, don’t differentiate pricing between APIs and databases, and can’t be leveraged to replicate high-volume databases. They also often break.

So Airbyte’s Postgres connector was a good trade-off. With open source, our users could customize the connector to fully address their needs. Connect Postgres to Elasticsearch and hundreds of other destinations, no matter the type.

This is the power of open source: Airbyte enables any users to address all their connector needs in one single platform, however custom or long-tail they are. That said, leveraging an alpha connector is not ideal. Having a connector that works for every use case off the shelf would be best!

A 3-month dedicated focus to bring Postgres to GA

Airbyte’s mission is to solve data integration by addressing every custom and long-tail connector need. But, before we focus on getting 500 or 1,000 connectors onto the platform, we need to get fundamentals right. We defined a quality certification checklist, as well as a clear roadmap for bringing the most important connectors from Alpha to Beta to Generally Available (GA).

What does GA mean for Airbyte? It means support for EL-related features - all streams supported, incremental syncs or Change Data Capture (CDC) when possible - along with a 99% SLA and live support.

This is a hard task when building API connectors and an extremely complex task for database connectors. The fact is, every company has high-volume databases that require increased security as they manage production data, and often have edge cases in the way the infrastructure is set up. So we knew this was a problem worth solving, especially given there aren’t great solutions in the market.

For the past three months, we had our database team (yes, we now have a dedicated team for database connectors) focused solely on Postgres.

What can you do with our GA Postgres connector?

Full syncs: copy an entire database from source to destination at once
Incremental syncs: send all data since the last sync, looking at changes that happen at the source
Change data capture (CDC): incremental syncs leveraging Postgres CDC

Learn more in our Postgres documentation.

What’s coming soon?

Support for schemas for databases with 10K+ tables
Automated schema evolution, currently limited to adding/removing tables or resetting a single table
Column selection

We will also be releasing custom connector support to Airbyte Cloud (already available in Airbyte Open Source) in the next quarter to give users the ability to customize this GA connector as needed.

Why this Postgres GA is a game changer.

It works reliably and at scale. Table stakes but important. Airbyte also has:

Live support in case you run into any issues. Our Airbyte Cloud live chat response time is between five to ten minutes on Slack and Discourse
A thriving community on our Forum for any Airbyte Open Source or Cloud questions that is easily searchable and being monitored by our Community Assistance Team 24/5
Pricing that starts at $10 per GB and discounts for bulk credit packages. The higher the package, the bigger the discount. Talk to sales if you want to learn more. (pro tip) Under our new volume-based pricing, Airbyte Cloud is actually three times cheaper than alternatives for a 100GB database, and 40% cheaper for a 1TB database.

Our team is happy to create an unlimited 30-day free POC so you can fully test our Postgres connector and get precise estimates about its costs and value to your organization.

What’s next?

We learned a ton about database connectors in the past three months and plan to apply those learnings to the next round of connectors. We’re bringing our MySQL and MSSQL source connectors to GA within the next two months. Check out our roadmap for more!

If you’re interested in helping solve data integration for all, join our maintainer program. Our current focus is 100 GA connectors, then we’ll focus on getting to 1,000+ connectors!

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 14-day free trial