Blog

Data

Airbyte 2.2: What to Know About the New Release

Explore Airbyte 2.2, including its latest features, improvements, and key updates for building more reliable and efficient data pipelines.

DATA

Richelle Herrli

Summarize with AI:

Airbyte 2.2 shipped for self-managed instances on August 10 and it has some major Helm upgrades for you to keep in mind. You'll also notice security bug fixes and a job history optimization.

Check out the major highlights of this release before you upgrade to get the most out of the new version and avoid some obvious pitfalls.

If you still use Self-Managed Enterprise, do not upgrade. Speak to your Airbyte representative about switching to one of Airbyte's Cloud plans, which always have the latest capabilities and security improvements. Private Link is available if you require it.

1. Airbyte 2.2 improves enforcement of Helm chart V2 global defaults

As of Airbyte 2.1, you can only use Helm chart V2, which ignored global.jobs.resources. If you set a CPU or memory limit for sync jobs there, the chart ignored those limits and ran on defaults. Plenty of operators never noticed because a silently ignored limit acts like a generous limit.

In 2.2.0 those values take effect as fallbacks for the following keys:

JOB_MAIN_CONTAINER_CPU_LIMIT
JOB_MAIN_CONTAINER_CPU_REQUEST
JOB_MAIN_CONTAINER_MEMORY_LIMIT
JOB_MAIN_CONTAINER_MEMORY_REQUEST

Keep in mind that keys scoped under global.workloads.resources.mainContainer take precedence; anything you leave unset falls back to the legacy value.

The same reactivation applies to global.workloads.resources.replication.*, which now sizes the replication orchestrator appropriately. You'll need to check the following values:

REPLICATION_ORCHESTRATOR_CPU_LIMIT
REPLICATION_ORCHESTRATOR_CPU_REQUEST
REPLICATION_ORCHESTRATOR_MEMORY_LIMIT
REPLICATION_ORCHESTRATOR_MEMORY_REQUEST

This separation helps you control the orchestrator and main container independently to account for their different workloads and appetites.

Take stock of these limits before you upgrade. A harmless value in 2.1.x can easily become a scheduling constraint on the first post-upgrade sync. If you tuned around the broken behavior by adding limits somewhere else, that limit will compound.

2. The Helm chart renders cleanly through your deployment tooling

If you pipe helm template output into Kustomize, or run any GitOps flow that renders manifests before applying them, 2.2 offers a couple of fixes.

The chart no longer emits duplicate DATAPLANE_CLIENT_ID_SECRET_KEY or DATAPLANE_CLIENT_SECRET_SECRET_KEY, and the Temporal UI deployment's resource and security context blocks are indented correctly. Kustomize will outright refuse duplicate keys and bad indentation, so this can take your work from a YAML with script patches all the way to a working pipeline.

Both the main chart and the data plane chart now accept a top-level namespaceOverride. namespaceOverride controls the namespace for two things:

Chart-generated resources
Internal service discovery

Rendered manifests always carry an explicit metadata.namespace set to namespaceOverride if configured, or the release namespace otherwise. That manifest namespace takes precedence over a kubectl -n call, so render your templates with a --namespace that matches the namespace you're deploying into.

3. HTTP Basic is no longer accepted

In self-managed Airbyte 2.2, any script, cron job, or CI step that calls the API with curl -u will break after the upgrade.

Instead, you'll need to request an access token from /api/v1/applications/token using instance-admin-client-id and instance-admin-client-secret. There may be more calls using HTTP Basic auth than you'd expect because it's an easy health check.

Two related fixes to note. Unauthenticated requests no longer advertise a Basic authentication challenge, which caused browsers to surface a native username and password dialog for anyone who hit the URL.

The server now also strips client-supplied internal authentication headers from incoming requests before deriving the trusted workspace and organization identity headers for authorization. This means Airbyte is less likely to accept a spoofed request.

The same updates apply to administrative invitation endpoints. Responses from /v1/user_invitations/create, /v1/user_invitations/list_pending, and /v1/user_invitations/cancel no longer include inviteCode, so listing pending invitations doesn't include working invitation codes. cancel still accepts inviteCode as input.

4. No more false reporting on sync failures

Five of the fixes in this release share a similar theme. The sync was fine and the platform either reported a failure or destroyed the working state on the way out.

A sync that produces zero records used to fail. The container orchestrator could terminate a destination before the destination had a chance to shut down gracefully, and it missed exit-code files created through an atomic rename. Destinations now get up to 60 seconds to exit on their own before Airbyte stops them, and empty syncs complete normally. If you run incremental syncs on slow-moving tables, you should see fewer false alarms.

For connections on the socket transport, the workload launcher read CPU limits expressed in millicores as whole cores. A limit of 500m was read as 500 cores, and the computed socket count inflated enormously. That's now parsed correctly.

There's a big OAuth fix in this release. When a source emits a CONTROL message with refreshed OAuth configuration, Airbyte now saves the information earlier in the process. Before, cancelling or aborting a sync after the connector refreshed its token resulted in a lost rotated refresh token.

Two changes push errors to surface immediately. If Airbyte creates an attempt on a job that's already running or finished, the call returns a 409 Conflict so the workflow fails fast instead of retrying. Additionally, secret storage configurations for AWS, Azure, GCP, and Vault are validated for required fields when you create them through the API. Before this release, you might successfully configure secret storage only to have it fail during a secret operation.

5. Job history queries get faster

On large installations, the job history queries have made the UI crawl. To speed up these queries, Airbyte 2.2 does the following:

Adds indexing on jobs(created_at) and on the commands workload lookup
Gives job pruning and connection-timeline event pruning a ten-minute statement timeout
Rewrites attempt statistics to use bound array parameters and chunked queries

You'll notice the cost for this update during the upgrade. Those indexes are created concurrently outside a transaction. For this reason, you can expect a large jobs table to sit in that step for a long time during startup. Plan your maintenance window accordingly rather than assuming the pod is wedged.

One more thing in the same spirit: a failed schema discovery on the refreshed-catalog path no longer blocks the connection page from loading, so you can actually reach the connection and fix the source of the problem instead of being locked out.

Housekeeping notes before you upgrade

This upgrade is one-way. The 2.2.0 migrations drop tables that 2.1.x depends on, so a database that has run the migration can't serve 2.1.x images, and nothing dropped is recoverable. Take a backup.

Then, in order:

Grep values.yaml for global.jobs.resources and global.workloads.resources.replication.*. Confirm the values are what you want or remove them.
Find every API caller using curl -u and switch it to an access token. You can get this token by following the steps in Airbyte documentation.
If you mirror or pin bundled images, mirror manifest-server tag 7.23.7. Pods won't start without it.
Schedule enough of a window for concurrent index creation on your jobs table.

Find the full details in the 2.2 release notes.

About the Author

Richelle Herrli

Richelle Herrli is a Senior Developer Advocate at Airbyte

Try Airbyte Agents

Be among the first to explore our new platform and get access to our latest features.

Try it free

Airbyte 2.2: What to Know About the New Release

Try Airbyte Agents

Related posts