How Do You Design a Real-Time Analytics Architecture?

The core work lies in state recovery, schema changes, late data, and continuous infrastructure that must stay correct through traffic stalls and spikes. Regulated environments also constrain placement when consumers span cloud analytics and in-boundary systems. Make ownership, failure recovery, and data-boundary constraints explicit before launch.

TL;DR

• Start the design with the freshness tier the consumer needs; every component choice downstream follows from it.
• Sub-second freshness carries a higher operational floor than batch.
• Correctness comes from event-time semantics plus exactly-once delivery composed across the CDC connector, the log, the processor checkpoint, and the sink commit.
• Where the data plane runs and whether the components are open source decide portability, and whether regulated sources can join the pipeline at all.

Try Airbyte Flex

What Is a Real-Time Analytics Architecture?

A real-time analytics architecture chains four layers: ingestion, stream processing, serving, and consumption. Together, these layers allow queries to return within a stated freshness target. Ingestion usually uses Apache Kafka, with Debezium for CDC. Processing uses Apache Flink or Spark Structured Streaming. Serving uses an Online Analytical Processing (OLAP) store such as Apache Pinot or an Apache Iceberg table, while consumption occurs through a dashboard, a product API, or a warehouse. Two different things get called real-time: processing data in motion, and querying data that arrived recently. Fraud models need processing in motion, while people reading charts need recently arrived data.

Data plane flexibility lets you place the remaining components on premises or in the cloud according to their latency and cost requirements.

How Do You Set Freshness Targets Before Choosing Components?

Use real-time when an automated action depends on low latency; human consumers rarely need sub-second data. Treat real-time as a cost model rather than a feature.

Four questions come before any component list. What latency does the consumer's Service-Level Agreement (SLA) require? Do events arrive continuously or in bursts? What is the cost envelope? A broker, a consumer, and a state store run whether or not data flows, while batch can stop when idle. Does the team already carry schema evolution, broker tuning, and consumer lag monitoring? If the answers favor periodic processing, pick batch rather than adding streaming infrastructure by default.

The minutes band is where hybrid ETL pipelines using ETL, CDC, or batch replication into a lakehouse belong. This is the default for historical reporting and most Business Intelligence (BI) loads until an automated consumer proves otherwise. The sub-second tier requires continuously running ingestion, processing, and low-latency serving infrastructure.

Match the freshness tier your consumer needs to the lightest stack that reaches it.

Most organizations run a combination of tiers, which is why the next question is how each tier gets its data in.

How Should You Design Ingestion and Change Data Capture?

CDC from operational databases and a durable log are the two ingestion primitives; everything else connects into one of them. Debezium's core connectors can participate in exactly-once delivery, the first handoff in an exactly-once chain you still have to compose yourself.

Log-based CDC reads the database's Write-Ahead Log (WAL) or binlog instead of running bulk exports that can create production load or locking risks. Partition the Kafka topic by primary key so every change to one row lands in one partition, in order. That preserves per-row ordering and nothing more: one source transaction touching two tables can surface in the sink at different moments, so cross-partition ordering stays your problem.

Source location decides where the data plane has to run. For workloads that require electronic protected health information (ePHI), card data, or Call Detail Records (CDRs) to remain inside your boundary, keep the connector, its credentials, and its compute there. When sources span SaaS APIs and in-boundary systems, hybrid data integration lands both in the same log, with the capture leg staying beside the regulated source.

How Do You Handle Stream Processing, State, and Correctness?

Correct results on out-of-order data require event time with watermarks; processing time gives fast answers that can go wrong when the network stalls. Event time is when the event happened, while processing time is when your system saw it. The skew between them varies with source, engine, and hardware, so windows keyed on processing time miscount without complaint.

A watermark estimates event-time progress and marks the point before which the processor no longer expects on-time events. Records can still arrive after that point and must follow your configured late-data policy, and watermark latency can change as workers scale and data becomes unevenly distributed. A wider out-of-order tolerance catches more late data at the price of latency, so budget engineering time for tuning that tolerance per source.

Exactly-once has to compose across handoffs: Debezium's delivery into Kafka, the processor checkpoint that records consumed offsets and operator state atomically, and a sink commit tied to that checkpoint. No single component documents the full chain; you own that composition.

Flink and Spark Structured Streaming differ primarily in latency and operational requirements. Choose Flink for stateful workloads that require event-at-a-time processing, and choose Structured Streaming when micro-batch latency meets the requirement and you already run Spark. Flink often adds a separate operational surface for deployment, checkpoint tuning, and state backend configuration, while Structured Streaming reuses the batch engine you already operate. The workload that earns Flink's overhead is one whose consumer cannot tolerate the latency of micro-batching; if your consumer would be satisfied at two seconds, you may not have that workload.

Which Serving Layer Fits Your Query Pattern?

Choose the serving store according to query pattern and concurrency. Real-time OLAP stores answer ad hoc queries, streaming databases maintain supported continuous query models, and Iceberg holds history.

Pinot, Druid, ClickHouse, and StarRocks answer high-concurrency, high-cardinality queries on data at rest, a pattern that fits a dashboard with hundreds of simultaneous users. Streaming databases differ in the joins and continuously maintained query models they support, while real-time OLAP engines commonly support more open-ended exploratory queries. Plan the store for large data volumes on concurrency and ingest throughput first, then check freshness. Consumers decide the transport: dashboards poll, clients that cannot poll receive pushes over WebSockets or Server-Sent Events (SSE), and warehouses read the lakehouse on a batch cadence. Other downstream consumption patterns include retrieval stores and AI data products, which consume replicated outputs while staying outside the ingestion path.

Kafka or CDC can feed minutes-fresh Iceberg tables for the middle tier, and the sub-second band often uses a hybrid architecture where Iceberg anchors history while a hot layer serves the moment. Open table formats and processors reduce migration scope if you later change the hot store, though schemas, ingestion contracts, indexes, and query behavior may still require work. Version 3 of the Iceberg spec replaces V2 positional delete files with binary deletion vectors stored as Puffin files and adds row lineage fields, _row_id and _last_updated_sequence_number.

What Breaks in Production and How Do You Design for It?

Streaming failures are silent and continuous because a stream can keep emitting slightly wrong results until someone notices, while a batch job fails loudly once a night.

Alert on consumer lag, late-data counts, and checkpoint duration. That trio provides a starting recommendation with workload-specific scope. Route CDC upserts to Merge-on-Read (MOR) sinks rather than append-only ones. Checkpoint intervals and watermark tolerances are workload-specific settings, not universal benchmarks.

Kafka topics can carry row-level events or records, while Flink state backends store operator and keyed state used during stream processing, so treat data pipeline security as a launch gate. Require logging and lineage from connector offset to sink commit so a bad record traces to its source transaction.

Define a response for each way the production pipeline can break before launch.

Design responses for these failure modes before launch, and the same boundary questions that shape ingestion resurface once you decide where that pipeline physically runs.

How Airbyte Flex Keeps Analytics Data In-Boundary

Airbyte Flex covers the CDC ingestion leg into the lakehouse or warehouse serving layer as a hybrid deployment. Airbyte runs the control plane for orchestration and upgrades. Its in-boundary data plane keeps data and credentials inside your boundary and runs connector compute there.

That placement lets an ePHI or card-data source join the same pipeline as a SaaS source. Flex ships 700+ connectors, log-based CDC without production table locking, and open destinations including Iceberg, so the table format stays portable. Airbyte's open-source foundation makes the platform inspectable, with adoption across 7,000+ companies and 18% of the Fortune 500.

The same boundary holds once fraud-scoring or pricing agents start reading from this data. Because Flex keeps replication inside your environment instead of assembling context through vendor-controlled APIs at runtime, agent queries inherit the same governance and audit trail as the replication pipeline that feeds them.

Where Should You Start?

Start with each consumer's SLA, source location, and infrastructure cost, and let those three constraints narrow the design before you pick a single component.

Airbyte built Flex for minutes-band ingestion under data-boundary constraints, adding Role-Based Access Control (RBAC) and audit logging so the compliance constraint is covered alongside the latency one. That same boundary extends to the agents and AI products reading the data downstream.

Get a demo to see how Airbyte Flex lands CDC sources in a minutes-fresh lakehouse.

Frequently Asked Questions

When Should You Choose Lambda Over Kappa?

Choose Lambda when backfill and historical reprocessing matter as much as fresh results, including fraud detection and financial workloads. A pure Kappa design depends on log replay for reprocessing, which can consume substantial time and cluster resources at high data velocity. A modified approach can classify jobs and reprocess historical data from the warehouse.

How Do You Keep Multi-Table CDC Transactionally Consistent Across Iceberg Tables?

Per-table Iceberg commits do not preserve source transaction boundaries, so a join can see one table updated and the other not. You can group related tables under one job with a shared commit cadence, but this does not guarantee transactional consistency. Otherwise, you accept read skew bounded by the checkpoint interval.

What Happened to the Streaming Database Category?

Use query pattern to evaluate streaming databases. Systems that continuously maintain supported query models fit known workloads, while real-time OLAP stores fit open-ended, high-concurrency analysis. Evaluate the consistency model, supported joins, and serving pattern rather than relying on the category name.

What Does Iceberg Spec Version 3 Change for CDC Pipelines?

Version 3 changes how Iceberg represents deletes and tracks row lineage for CDC pipelines. Flink can convert equality deletes into deletion vectors on format version 3 or later. On AWS, support depends on running a current Amazon EMR or AWS Glue release, so check the release notes for the minimum version before you rely on it.