Load balancing Airbyte workloads across multiple Kubernetes clusters

Learn how Airbyte load balances data movement workloads across multiple Kubernetes clusters

Jimmy Ma

Jul 8, 2024

Updated Jun 29, 2026

•

5 min read

Summarize with AI:

Load balancing Airbyte workloads across multiple Kubernetes clusters

Airbyte is designed to handle data replication jobs and Airbyte Cloud comes with additional constraints from supporting a broader range of enterprise clients. Many clients are required to run workloads within a specific geographic region (Europe or US) for regulatory reasons and also as an opportunity to reduce infrastructure costs.

As part of supporting multiple regions we adopted a control-plane/data-plane architecture. The idea is that we have a control-plane that orchestrates workloads across multiple data-planes. Think of a control-plane as the brain of the operations, how do we configure workloads, when should they run and reporting to name a few examples. The data-planes are the muscles doing the heavy lifting a.k.a data movement. Technically speaking, each plane would be a Kubernetes cluster, and originally we had one data-plane per geographic region.

With our recent growth, we have had to deal with an ever increasing load which was starting to put pressure on our infrastructure. We ran through diverse issues, from our cluster being out of capacity to being rate-limited by the Kubernetes API. The latter pushed us to take a closer look at how we were interacting with the cluster. While there was room to improve our interactions with Kubernetes, it also made it clear that having many medium sized clusters was better operationally than building up to massive clusters.

In this article, I will share how we solved load balancing Airbyte workloads across multiple clusters.

Airbyte Kubernetes challenges to scale

Before diving into the details of load balancing, I will share some context to better understand our load balancing constraints such as the nature of our workloads as well as some Kubernetes behavior that impacted our design.

Let’s go over the workloads we need to schedule. The core of the problem is about scheduling the pods that will effectively move data: a source connector, a destination connector and an orchestration pod. They generally function as a cohesive triplet as data cannot move if one of the three is missing. At the Kubernetes level, what we call scheduling is finding a suitable node to run the pod. A node being close enough to a VM and the most granular unit of scaling of a Kubernetes cluster. In order to start a sync, we create those pods which means that Kubernetes will attempt to schedule each of those pods independently on the available nodes.

The hourly load spike challenge

Kubernetes scheduling goes hand in hand with the auto-scaler. The auto-scaling allows on demand addition or removal of nodes according to the load of the cluster.

With Airbyte, the load will typically be hourly spikes as it is a common frequency of data movement. Airbyte does not artificially spread jobs across the whole hour. As a result, our load is typically highest at the top of the hour. This caused a first scaling challenge as the default Kubernetes scheduling strategy is to spread the load across the available nodes. This maximizes the number of nodes being used, preventing cluster scale down and proper auto-scaling

A fairly straightforward fix here is to change the scheduling logic to bin-packing which in turns will end up minimizing the number of required nodes by assigning a pod to the most loaded node that has enough resources to host the pod. The bin-packing scheduling strategy enabled an efficient auto-scaling of the Kubernetes cluster.

Limited resources could cause deadlocks and job failures

Another element to consider is the impact of a cluster being at its resource capacity on our data movement jobs. When the cluster is overloaded, we are in a situation where pods are in a pending state, waiting for resources to free up so they can be scheduled. Typically, new pods can be scheduled as on-going jobs terminate.The system naturally recovers on short load spikes.

Under-provisioned clusters experiencing longer-than-usual job spikes ran into another issue: the incomplete sync pod triplet (source, destination, orchestration pod). The non-deterministic scheduling means it is likely to end up in a situation where a lot of syncs are waiting but never get all three pods scheduled. We are in a deadlock-like situation where multiple pods are waiting for the last member of the triplet which isn’t going to appear because the cluster is out of resources. If this happens, workloads will eventually timeout which means that jobs are failing.

This deadlock situation is one of the scenarios we want to minimize when thinking about cluster load balancing.

How we implemented load balancing across clusters

Load Balancing is a common challenge when designing infrastructure. A common example is load balancing web requests across a pool of application servers. Round robin, which spreads requests evenly, works well here as typical web requests are short lived and have fairly similar load on the server. Some variant of this might involve adding some weight factor if the underlying server has different specifications.

Intuitively, this model should be applicable to spreading load across multiple clusters. In practice, there is one subtle difference to consider: how long does it take for a request to be processed. A few operations are significantly slower. For example, scheduling our triplets on the cluster can easily take 40 seconds, while adding a node can take up to 5 minutes. At the top of the hour, we could have hundreds of jobs waiting to be scheduled. This means that we are effectively enqueueing jobs to be processed rather than dispatching short lived requests.

Working with a queue has different implications, for example, there could be a considerable lag between the time the job is enqueued and the time the job is dequeued. This lag is an important factor when it comes to cluster attribution because of the operational implications. A common issue to think about with queuing systems, is what happens when the consumer becomes unavailable. The processing of a job could fail if the consumer crashes. In this case, from where should the job be retried, should it be requeued by the queuing system or the producer? We will cover some of those questions as we go over some of the options we considered.

First class of options: Assigning from the control-plane

The first option was to assign a cluster from the producer. This solved the question of predictability: knowing where a workload will run enabled us to easily check on the outcome of a workload, whether it was healthy or disappeared for diverse reasons. On the other hand, this brings up the question of which cluster should we assign it to.

The hardest question cluster assignment brought up is resource awareness. Being resource aware meant being able to assign the cluster with the most available resources as opposed to a blind round robin for example.

Round robin is generally an option to consider. It is a decent solution especially given how easy it is to implement compared to its alternatives. A major flaw in our case is that in the event a cluster would be overloaded, we would keep on scheduling jobs for the out-of-resource cluster and this could put us in a deadlock state which we want to avoid.

Trying to avoid scheduling jobs to an overloaded cluster meant being able to assign workloads knowing the available resources on the cluster. This proves to be harder than it looks, as it implies that the control-plane would then be aware of the usage and capacity of each data-plane. Each cluster being configured with a node autoscaling, it also means that the max capacity needs to be registered as this configuration isn’t exposed through kubernetes interfaces. Furthermore, when running on cloud infrastructure, setting a maximum also doesn’t guarantee that we will be able to reach it as it may vary with the availability of a zone. Overall, this means that it is effectively hard to predict the actual max capacity of a cluster.

Another concern comes from the assignment itself. A cluster is bound to become unavailable, whether it is from a planned operation or some incident. When this happens, this means having to deal with draining the queue and rescheduling workloads. As a general practice, we prefer a system that can self heal. Needing an operator to perform queue related tasks is something we wanted to avoid. Even though most of this could be automated, that remains extra tooling to build and maintain.

Our solution: Inversion of Control

As we covered, assigning a cluster ahead of time comes with some challenges for picking the right cluster: what if we could avoid this problem altogether? The main idea is to shift the responsibility to the data-plane: only pick a job if the data-plane has enough capacity to process it.

The implications here is that we will set up multiple data-planes to read from the same queue and that each data-plane will be competing for the next available job.

This idea gives a different option to the resource capacity problem. When a pod is waiting for resources to free up on the cluster, it remains in a pending state. This effectively means that the cluster is at current capacity and behind the scene, kubernetes may try to scale up if possible. This is an interesting property because it means that we can leverage this to understand that we’re putting enough pressure on a given cluster and it might be time to backoff.

The consumers of the job queue become our control point, we can either decide on a number of consumers per data-plane. This becomes our backpressure mechanism as well as how to limit the blast radius when it comes to infrastructure errors. By controlling this number of consumers, we can limit the amount of overwork a cluster can commit to and avoid a never ending cycle of triplet start-up deadlock situations.

This inversion of control addresses the load balancing. In a situation where all the clusters have resources to process workloads, where it runs doesn’t really matter. On the other hand, if a cluster becomes overloaded, it will stop processing new messages, leaving them to the ones that have available resources since they will be the ones still consuming from the queue.

This also addresses the other concern about queue operations. The jobs are not being assigned until actually being worked on, this removes the need to be draining queues or rerouting jobs we had in the other approaches.

Closing notes

Our answer to load balancing workloads across multiple data-planes is to push down the responsibility to the data-plane itself. We enqueue workloads in a single job queue and let the data-planes compete for jobs to process if they have capacity to do so. This has the benefit of treating capacity as a problem that is local to a cluster, removes the complexity of planning ahead for available resources and keeps operations simple when faced with cluster downtime.

This inversion of control is a guiding principle that we kept in mind when designing Workloads which is the new component that is part of the airbyte-worker redesign.

Load balancing Airbyte workloads across multiple clusters contributes to Airbyte Cloud further fulfilling security, latency, and scaling requirements for enterprises.

What other great readings

This article is part of a series of articles we wrote around all the reliability features we released for Airbyte to reach the 1.0 status. Here are some other articles about those reliability features: