Pipelines & orchestration
Design, schedule, transform and monitor data pipelines, from batch ELT with dbt and Airflow to real-time streaming with Kafka.
- 489
- Resources
- 6
- Subtopics
- 9
- Formats
The essentials of pipelines
Streaming & real-time
10 explainers & guides
Transformation & dbt
11 explainers & guides
Orchestration
10 explainers & guides
- Building Data Pipelines with Apache Airflow
- Central Data Orchestration Regional Data Plane
- CI CD Pipelines
- Data Orchestration
- DataOps the Definitive Guide
- DevOps Pipeline
- How to Manage Dependencies and Retries in Data Pipelines
- Luigi Python Data Pipeline
- Multi Agent Orchestration
- The Difference Between Orchestration and ETL
Workflow automation
19 explainers & guides
- 7 Agentic Automation Tools For Context-Aware AI Workflows
- Accounts Payable Workflow Automation
- AI Recruitment Agents for Hiring Automation, Screening, and Workflow Design
- Benefits of Workflow Automation
- Cloud Workflow Automation
- CRM Workflow Automation: From Static Rules to Autonomous Agents
- Data Automation
- Data Pipeline Automation
- Document Workflow Automation
- Enterprise Workflow Automation
- Healthcare Workflow Automation
- Legal Workflow Automation
- Low Code Workflow Automation
- Marketing Automation Workflow
- Outbound Automation
- Pipeline Automation
- Real Estate Workflow Automation
- RevOps Automation
- Workflow Automation
Python for data
8 explainers & guides
Pipeline design
30 explainers & guides
- AI Data Pipeline
- Azure Data Pipeline
- Bitbucket Pipelines Set Up Guide
- Build a Data Pipeline
- Build ETL Pipeline PostgreSQL Tableau
- Building ETL Pipelines in Databricks
- Catch Null Invalid Data in Pipeline
- Data Ingestion Pipeline Best Practices for Modern Organizations
- Data Pipeline
- Data Pipeline Architecture
- Data Pipeline Management
- Data Pipeline Observability
- Design ETL Pipelines Hybrid Cloud Environments
- ETL Pipeline
- ETL Pipeline Bottlenecks Causes Fixes
- ETL Pipeline Pitfalls to Avoid
- ETL Pipelines Run
- External Secrets Management Data Pipelines Security Guide
- Handle Schema Changes without Breaking ETL Pipeline
- How Do I Monitor ETL Pipeline Health?
- How to Write Test Cases for ETL Pipelines a Beginners Guide
- Hybrid Data Pipeline Security Exchange CVE Lessons
- Idempotency in Data Pipelines
- Natural Language Processing Pipeline
More on pipelines, by format
Tutorials & stacks
37 resources
- Incremental Change Data Capture CDC Replication
- Customer Segmentation Analytics Stack with Shopify Airbyte dbt Dagster and BigQuery
- Data Pipeline with dbt DuckDB
- Python ETL in Docker
- How to Use Airflow and Airbyte Together
- Building an E-Commerce Data Pipeline a Hands on Guide to Using Airbyte dbt Dagster and BigQuery
Put it into practice
Connection guides & Python pipelines
36 connection guides and 257 PyAirbyte pipelines relate to this topic.
Destinations
Your next pipeline
starts here.
Put your knowledge to work. Connect 700+ systems to your inference and analytics layers.