Blog

Engineering

Understand and Troubleshoot Your Migration to abctl

Understand and Troubleshoot Your Migration to abctl Airbyte

ENGINEERING

December 5, 2024

10 min read

Marcos Marx

Summarize with AI:

Understand and Troubleshoot Your Migration to abctl

Introduction

Airbyte recently introduced abctl as the new way to deploy Airbyte OSS, replacing the previous Docker Compose deployment method. This transition moves Airbyte to a Kubernetes-based architecture, providing better scalability, resource management, speed, performance and easy migration from abctl to a production Kubernetes.

This guide addresses common migration challenges and solutions based on user experiences.

Best Practices for Successful Migration

Pre-Migration Preparation
- Document all current configurations
- Back up all data and configurations
- Ensure sufficient system resources
- Test in a non-production environment first
Installation Environment
- Use dedicated machines/VMs
- Ensure clean environment without conflicting services
- Meet minimum resource requirements
- Set up proper monitoring
Post-Migration Verification
- Verify all connections work
- Test data syncs
- Monitor resource usage
- Document new configuration for team reference

From `.env` to `values.yaml`

The transition from .env files to Kubernetes-Helm-style values.yaml configuration represents a significant change in deployment management.

Airbyte leverages Helm as its package manager for both abctl deployments (PoCs and single-node) and large Kubernetes clusters. This unified approach simplifies scaling from development to production environments.

For comprehensive understanding:

Review Helm Value Files documentation
Explore Airbyte configurations in Artifact Hub, where you can find all available versions and compare configurations

Consider this practical example:

# Database Configuration
DATABASE_USER=airbyte
DATABASE_PASSWORD=your_password
DATABASE_URL=postgresql://airbyte:your_password@my-cloud-data.cloud.com:5432/airbyte
DATABASE_DB=airbyte

‍

The new configuration consolidates all deployment settings into a single versioned file:

global:	
  database:	  
    type: "external"	  
    host: my-cloud-data.cloud.com	  
    port: 5432	
    database: airbyte
    userSecretKey: database-user	
    passwordSecretKey: database-password

‍

This example demonstrates external database configuration with Airbyte. The userSecretKey and passwordSecretKey parameters enhance security by managing credentials through Kubernetes secrets rather than exposing them in environment files.

Apply your configuration using:

abctl --values PATH/TO/values.yaml

‍

When should you use the values.yaml during the migration from Docker Compose to abctl?

Whenever you have customized your .env using Docker Compose, such as for an external database, logging, custom URLs, etc.
Please check the latest version of values.yaml in Artifact Hub.

Frequently Asked Questions During Docker-Compose Migration and Solutions

Resource Requirements and Performance

Q: Why does abctl require more resources than Docker Compose?

abctl runs a Kubernetes-on-Docker single-node cluster (via Kind), which has higher baseline resource requirements

Straighforward test from dev to production infra configuration
abctl uses Helm Value chart which will keep history of changes
This allow dev deployments to run the same structure as a huge production cluster. Making simpler to have dev-prod with same variables just changing number of workers.
Drawback: uses more resource for small deployments.

Solutions:

Review minimum requirements in documentation
Consider -low-resource-mode for testing environments only
Scale machine resources based on number of concurrent parallel syncs
Monitor ongoing discussions about resource optimization

Q: How can I handle resource-related installation failures?

Kubernetes enables granular resource configuration per service and job. Each connection sync requires 2-3GB cluster memory. For example, three concurrent connections need 6GB minimum, plus core service requirements.

If you need to change the default values you can edit the values.yaml file:

global:
	database:
	  type: "external"
	  host: my-cloud-data.cloud.com
	  port: 5432
	  database: airbyte
	  userSecretKey: database-user
	  passwordSecretKey: database-password
	jobs:
	  resources:
	    requests:
	      memory: 1G
	      cpu: 250m
	    limits:
	      memory: 4G
	      cpu: 1

‍In this example, the request will reserve and block this to be applied to the jobs service. It sets a limit on what the job service can run until it encounters an out-of-memory issue.

Troubleshooting and Debugging

If you need to check if Airbyte is running you'll need an additional tool called kubectl This tool allows you run commands in a Kubernetes cluster.

Please check official documentation here to install kubectl it is a single command.

Configure kubectl to run commands using abctl cluster as default

KUBECONFIG=~/.airbyte/abctl/abctl.kubeconfig kubectl config view --flatten > ~/.kube/config

‍

Get deployment status

Check deployment status and running services (pods):

kubectl get pods -n airbyte-abctl

‍

Example output:

NAME                                                              READY   STATUS            RESTARTS   AGE
airbyte-abctl-airbyte-bootloader                                  0/1     Completed         0          5m9s
airbyte-abctl-connector-builder-server-8545f64bd6-rjx44           1/1     Running           0          4m15s
airbyte-abctl-cron-7c698c9bd4-j79hz                               1/1     Running           0          4m15s
airbyte-abctl-pod-sweeper-544d944547-9b4rv                        1/1     Running           0          4m15s
airbyte-abctl-server-55b6f75cb4-tb8bq                             1/1     Running           0          4m15s
airbyte-abctl-temporal-595b6d87c7-v7n56                           1/1     Running           0          4m15s
airbyte-abctl-webapp-6c4c7dc8c8-f484p                             1/1     Running           0          4m15s
airbyte-abctl-worker-5f699675dc-m7k69                             1/1     Running           0          4m15s
airbyte-abctl-workload-api-server-7bd586b4b4-nr8xx                1/1     Running           0          4m15s
airbyte-abctl-workload-launcher-5c5c6d4fc5-zffqv                  1/1     Running           0          4m15s
airbyte-db-0                                                      1/1     Running           0          5m9s
airbyte-minio-0                                                   1/1     Running           0          5m9s

‍‍

Investigate Pods

If you have any pod in an status different from Running or Completed it is worthy do some investigation:

kubectl describe pod airbyte-abctl-airbyte-bootloader -n airbyte-abctl

‍

It will print a very long output but the most important (usually) is the last part called Events:

Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  6m35s  default-scheduler  Successfully assigned airbyte-abctl/airbyte-abctl-airbyte-bootloader to airbyte-abctl-control-plane
  Normal  Pulling    6m34s  kubelet            Pulling image "airbyte/bootloader:1.2.0"
  Normal  Pulled     5m58s  kubelet            Successfully pulled image "airbyte/bootloader:1.2.0" in 20.063s (36.644s including waiting)
  Normal  Created    5m58s  kubelet            Created container airbyte-bootloader-container
  Normal  Started    5m58s  kubelet            Started container airbyte-bootloader-container

‍

In events you're going to see what maybe is blocking your pod to run properly.

How to get pod logs

Run the following command

kubectl logs airbyte-abctl-server-55b6f75cb4-tb8bq -n airbyte-abctl

‍

Shows you the server logs.

How to access the pod terminal

kubectl exec --namespace airbyte-abctl -it airbyte-minio-0 -- bash

‍

It will create a interactive session with minio pod.

Operational Considerations

Q: How do I manage my abctl installation?

Starting/stopping services
- abctl doesn't have start or stop command:
  - You can run abctl install to start or install latest version or specificied version using --version flag.
  - When you run abctl uninstall it will delete the local cluster but not the data. So if you run abctl install it will create a new cluster but reuse your previous data.
Updating configurations
- Always check the Artifact Hub when you're upgrading abctl with custom values.yaml as it can change a configuration parameter.

Conclusion

While abctl represents a significant improvement in Airbyte's deployment architecture, the migration process requires careful planning and consideration of resource requirements. Understanding common issues and having proper troubleshooting tools ready will help ensure a successful migration.

We're committed to continuously improving abctl based on your feedback. Have you recently migrated to abctl? Please take a moment to share your experience by completing our abctl survey. Your insights will help us enhance the migration process and create better documentation for the community.

Join on Airbyte Slack to get help, share your experiences, and connect with other Airbyte users who have successfully migrated to abctl.

About the Author

Marcos Marx

Marcos is a support engineer at Airbyte.

Try Airbyte Agents

Be among the first to explore our new platform and get access to our latest features.

Try it free

Understand and Troubleshoot Your Migration to abctl

Try Airbyte Agents

Related posts