Financial Data Integration: Architecture for Banks and Fintechs
Source systems, CDC and streaming, lineage and residency controls under DORA, BCBS 239 and SOX, plus build vs buy for financial data integration at banks.

Financial data integration is the controlled extraction, normalization and movement of core banking, payments, CRM and market data into one analytics layer through batch ETL, log-based CDC, event streaming or APIs, with security, audit trail and data residency controls attached so the result holds up under regulatory scrutiny. For that work, pipeline latency is the wrong first metric: supervisors test whether each reported figure traces to its source, reconciles to the ledger and lands inside the reporting window, and a streaming platform answers none of those questions on its own. For ledger-fed regulatory reporting, scheduled CDC with lineage and reconciliation captured inside the pipeline beats a low-latency design, while streaming earns its cost on narrow workloads such as fraud scoring.
Summary
- Default core ledgers to scheduled log-based CDC and reserve streaming for decisions made while a payment is in flight.
- Write log positions, row counts and control totals at extraction and load, inside the pipeline run itself.
- Settle where records, credentials and keys may live before shortlisting vendors, because residency rules fix the deployment model.
- Hash or tokenize card numbers and personal fields during the sync so raw values never land in analytics tables.
- Include engineer hours spent maintaining connectors alongside license costs in the build versus buy calculation.
- Plan any legacy ETL exit around parallel runs, daily reconciliation and a documented point of no return.
Learn more about Hybrid Deployment
What Is Financial Data Integration?
Financial data integration is ELT with a burden of proof: the destination tables feed reports a supervisor can challenge, so every value needs a provenance record, a reconciliation to the ledger and a known processing location. Copying rows into a warehouse without that evidence is ordinary ELT.
Under the Basel Committee’s risk data principles, risk reports require reconciled and validated reports and aggregation that runs on a largely automated basis to minimize errors. A pipeline that moves data quickly but cannot prove what it moved has done half the job, and the half it skipped is the one examiners ask about.
How Does Financial Data Integration Work?
Your team extracts data from source systems, transports it and normalizes it into a shared model, capturing lineage as each stage runs. Lineage reconstructed afterward from job logs and wiki pages cannot show what a value looked like between stages, and that intermediate state is exactly what a reconciliation break forces you to inspect.
Source Systems Feed Ledgers, Payments, CRM and Market Data
Core banking ledgers are the hardest source because the oldest ones store data in formats only their owning programs understand. On Db2 for z/OS, a table must carry the DATA CAPTURE CHANGES attribute before Db2 writes the full before images a log reader needs. VSAM files have no integrated change log at all, so capture depends on how applications update them, for example through the CICS logger, which means your team has to settle the capture method for each ledger before extraction can begin.
Your team also needs the COBOL copybook to decode EBCDIC records with COMP-3 packed decimal numeric fields. Record the copybook version in lineage: a balance decoded with last quarter’s layout is a wrong number that looks right. Newer cloud cores expose REST APIs, webhooks, event feeds and bulk backups instead, which trades decoding problems for rate limits, idempotency keys and replay handling.
Payment data adds message standards that changed under banks’ feet. The MT and ISO 20022 coexistence period ended on 22 November 2025 for cross-border payment instructions; eligible remaining MT instructions undergo chargeable contingency conversion into ISO 20022 without the structured, rich data, so converted records reach your analytics layer less structured than native ISO 20022 messages.
The announced November 2026 deadline called for rejecting unstructured postal addresses in CBPR+ payment messages. Your historical MT addresses and new structured ones still need to coexist in analytics tables.
Domestic rails keep older formats: a NACHA ACH file is fixed-width ASCII with 94-character records, wrapped in file and batch control records that carry their own totals. Card clearing arrives as network-specific batch files, so preserve each layout and its control records during ingestion.
CRM, ERP and billing systems expose their data through SaaS APIs and are the most likely to change without notice. Their vendors add fields, deprecate endpoints and throttle requests on their own release schedule, so the integration risk is drift, not access.
Market data adds licensing constraints to those format concerns. Vendors deliver real-time feeds alongside scheduled end-of-day and bulk files, and your team must apply the license’s storage and access restrictions. NYSE, for instance, lets vendors store real-time data for internal purposes but not redistribute it externally later without a specific license. A market data table in your lakehouse is a licensed asset, and the entitlement metadata has to land with it or the first dashboard shared with a client becomes a contract breach.
Integration Methods Move Data by Batch, CDC, APIs and Webhooks
Batch extraction is the simplest method and still the right one for many sources. Batch ETL reshapes data before loading it; ELT loads raw records first and models them in the warehouse. ELT has an audit advantage, because the raw landed table is itself evidence of what the source sent. Full and cursor-based extracts need no log retention or special privileges, but a cursor on an updated-at column never sees a hard delete, and in a ledger a delete can be a reversal.
To capture the deletes a cursor misses, use log-based CDC to read committed inserts, updates and deletes from the database transaction log. It catches deletes and intermediate changes, puts little load on production tables, and requires log retention long enough to cover any sync outage plus the privileges to read the log. Trigger-based capture is the older alternative; it writes change rows inside each transaction, which adds work to the hot path a core banking DBA is paid to protect. Query-based polling sits between the two: easy to set up, blind to deletes and to any change that happens twice between polls.
APIs cover partner data and anything a bank exposes to fintechs, but regulatory changes leave access terms unsettled. The EU’s PSD3 and Payment Services Regulation reached a provisional political agreement on 27 November 2025, with formal adoption still pending in the subsequent legislative process, so your team cannot yet treat API access terms as settled.
In the US, a federal court enjoined the CFPB on 29 October 2025 from enforcing the Section 1033 data rights rule until its reconsideration is complete. Your team therefore needs bilateral data agreements that specify what it can pull.
For those agreed calls, API gateways handle authentication and throttling. Webhook receivers need idempotent processing and a replay path; retain event history explicitly for downstream analytics.
These methods map onto the tool categories a bank evaluates. Replication platforms such as Airbyte run scheduled batch and log-based CDC syncs from databases and APIs into a warehouse or lakehouse. Legacy ETL suites such as Informatica, IBM DataStage, SSIS and Talend (now part of Qlik) reshape data before loading, so they already hold mapping logic that a migration has to carry across. iPaaS products such as MuleSoft connect operational applications to each other through APIs rather than loading analytics tables. Choose each tool by which part of the data flow it owns, because an iPaaS flow that updates a CRM leaves no raw landing table to serve as audit evidence, and leaving a legacy ETL suite means moving its mapping logic, not only its connections.
Normalization Maps Inconsistent Formats Into a Shared Model
During normalization, your team changes values and risks losing their audit trail. Each source ledger codes accounts its own way, so keep a versioned mapping table from local codes to the group chart of accounts in the pipeline, and record the mapping version on every normalized row. Retain the original amount and currency, converted amount, rate source and rate date so your team can re-derive converted figures, and convert timestamps to UTC while preserving the original offset. For trading data the clock is regulated: from 2 March 2026, Delegated Regulation 2025/1155 requires business clocks synchronized to UTC, with high-frequency trading held to 100 microseconds maximum divergence.
Reconciliation checks close the loop: row counts and control totals per run, balance comparisons against the general ledger, and break reports routed to an owner. Under MiFIR transaction-reporting rules, firms must reconcile front-office trading records against submitted transaction reports when competent authorities do not provide data samples.
Extraction must preserve the source, log position and extraction time so your team can trace a reconciliation break through the mapping version and rate used in normalization. Without that extraction metadata, you can detect a mismatch but cannot establish where it entered, which leaves the supervisory question unanswered regardless of delivery speed.
When Do Banks Need Batch, CDC or Real-Time Financial Data Integration?
Banks need streaming only where a decision has to happen while a transaction is still in flight; ledgers feeding regulatory, risk and finance reporting are better served by scheduled CDC with reconciliation attached to every run. The five methods trade latency against cost and audit evidence in different proportions:
Fraud detection is the clearest case for streaming, and even there batch does part of the work. A fraud platform can score transactions in real time while feeding its controls from three kinds of features: computed at decision time, precomputed near real time and cached, and batch-calculated periodically. Your team can separate the immediate scoring path from the history used to calculate its features.
Regulatory reporting has the opposite profile. The ECB’s risk reporting guide expects complete lineage at data attribute level from capture through extraction, transformation and loading, and documents monthly risk reports taking 40 or more working days because of reconciliation errors, manual adjustments and weak data quality controls.
Those reporting delays require work on controls as well as transport. The Basel Committee’s January 2026 newsletter on BCBS 239 implementation identifies timely, accurate ad hoc reports as a continuing hurdle across fragmented data estates with manual and automated processes. Your team should address reconciliation and traceability before paying for lower delivery latency.
What Breaks When Banks Integrate Legacy Core Systems and Regulated Data?
Legacy cores, fragmented silos and regulated data break pipelines in predictable places, and each one needs a control built into the integration rather than bolted on. Supervisors are not describing a solved problem: the ECB’s 2024 targeted reviews found weaknesses in data governance, data infrastructure and IT architecture, and data accuracy and integrity, along with shortfalls against BCBS 239. Silos persist because every acquisition and product line brought its own ledger, and each one is a separate extraction problem with a separate owner.
Mainframe extraction is constrained by capacity and by format: large unloads compete with the overnight batch window, and mainframe software is commonly billed on peak processor consumption, so a careless full extract can cost more than the analytics it feeds. Log-based capture moves the read off the tables, which is why it is the default for Db2. IMS and VSAM need mainframe-native tooling: IBM’s replication product reads active and archived IMS logs, for example. Evaluate Airbyte source support separately for IMS, VSAM, Oracle, IBM Db2 and SAP HANA before pairing mainframe capture with downstream replication; that split is an honest cost of keeping the mainframe.
Schema drift is the slower break. A core banking upgrade renames a field, a SaaS vendor adds a column, and the pipeline either propagates the change, blocks it or fails. Under SOX, any of those outcomes is a change to a system feeding financial statements, so the drift policy itself needs documented approval and a change log.
Each regulation lands on a specific pipeline control:
Your team must choose pipeline locations that meet the applicable residency rules. India’s central bank requires payment system data to be stored only in India, including full end-to-end transaction details.
Location controls also extend to recovery infrastructure: Turkish banks must host primary and secondary systems domestically, including backups as secondary systems. Use regional pipelines landing in in-country storage, and assess whether aggregates or tokens may cross into a global layer.
Your provider assessment must accompany that location decision. The ESAs’ list of critical ICT providers under DORA names Amazon Web Services, Google Cloud, Microsoft and Oracle among 19 firms. Resolve deployment and provider oversight before choosing the downstream architecture.
What Does a Reference Architecture for Financial Data Integration Look Like?
The reference architecture assigns one method per source class, lands everything raw in a warehouse or lakehouse, and models upward through normalized and reporting layers, with run evidence written at every hop. The per-source pattern looks like this:
Your team must retain run evidence through destination maintenance. In Apache Iceberg, expiring snapshots removes old table versions from time travel, so a storage cleanup job can delete evidence without touching current rows.
To preserve the required history, use tagging snapshots for auditing to keep chosen versions past expiry. Assign an owner to review those retention settings before cleanup runs.
Build versus buy turns on connector upkeep, and you can price it with three numbers: connectors in scope, engineer hours each needs per month for API changes, schema drift and failed runs, and a loaded hourly rate. Using database architects as a salary proxy, their median annual wage of $139,500 in May 2025 puts a tenth of a year’s connector maintenance at about $14,000 before benefits and on-call. If twenty sources each need four hours a month, that is 960 hours a year, close to half an engineer, spent keeping existing data flowing rather than building anything new.
For a modeled composite organization, Airbyte delivered 239% ROI in Forrester’s Total Economic Impact study. Use your own source count and maintenance assumptions for the purchasing decision.
The options compare on cost driver, upkeep owner and residency fit:
Buy the replication layer and build the reconciliation rules, account mappings and lineage contracts yourself. The connectors encode other vendors’ APIs and change whenever those vendors ship; the mappings encode your bank’s accounting, and nobody outside the bank can maintain them correctly. Building connectors still makes sense for a proprietary mainframe extract or an in-house payments engine, which is exactly where a connector framework earns its keep.
How Does Airbyte Help With Financial Data Integration?
Airbyte covers the replication layer of this architecture and leaves the reconciliation rules and account mappings with you. Its 700+ connectors include log-based CDC that replicates database changes without locking production tables, run as scheduled syncs. Your team sets the policy for detecting and propagating schema changes, so drift can follow your review process.
For sensitive fields, hashing and encryption run during the sync before card numbers or personal values land in the destination. Use SSO, RBAC and audit logging to control and trace connection changes.
For residency, Airbyte Flex splits the deployment: Airbyte operates the control plane while the data plane, along with your records, credentials, keys, and compute, runs in your environment, though some metadata such as cursor and primary-key values sits in the control plane. Choose cursor and primary-key fields whose values are permitted in the control plane. Each workspace maps to one region and one data plane, so EU and Indian data get separate workspaces. The same governed tables can then feed AI applications under the same access controls and lineage as your reporting.
Why Should Banks Design Audit Trails Before Choosing Pipeline Speed?
Make recoverability part of pipeline acceptance before paying for faster delivery. Interrupt a sync after a partial write, then compare source and destination records after recovery. Approve the pipeline only after your team can explain the recovered result; a successful run alone does not prove that the reporting evidence survived.
Airbyte gives that design a replication layer you can inspect and version. Its open-source connectors let your security team review the code that touches regulated data, the Terraform provider keeps connection configuration in source control as change-management evidence, and the API lets Airflow, Dagster or Prefect trigger syncs so each run sits in the same orchestration history as your reconciliation jobs.
Get a demo to see how Airbyte keeps governed replication inside your infrastructure boundary.
Frequently Asked Questions
How is data analysis used in finance once data is integrated?
Integrated data feeds risk aggregation, product and customer profitability, customer analytics and regulatory reporting from one set of reconciled tables. The benefit is consistency: the risk team’s exposure figure and finance’s balance come from the same rows, so differences between reports point to definitions rather than to two pipelines disagreeing.
Does financial data integration require a data lakehouse?
No, a warehouse alone works well when most sources are relational and the main consumers are SQL reporting tools. A lakehouse fits better when you hold large volumes of tick data, documents or model features, or want several engines reading the same tables. Open table formats such as Iceberg keep that choice reversible, because the data stays readable by more than one engine.
Who should own financial data integration at a bank?
Ownership splits three ways. The platform team owns connectors, sync schedules and infrastructure; finance owns account mappings and reconciliation sign-off; risk owns aggregation definitions and data quality thresholds. Problems start when the platform team is asked to approve accounting logic it cannot judge.
How long does migrating off legacy ETL usually take at a bank?
Longer than the tool swap suggests, because the old platform stays the system of record until the new one proves it produces the same answers. Plan for a coexistence period with parallel runs and daily reconciliation of reports, migrate in waves with exit criteria per wave, and document the point after which rolling back is no longer safe.
Can fintechs use the same integration architecture as large banks?
Yes, the pattern carries over, with fewer sources and lighter regulatory scope at the start. A fully managed service suits a fintech whose data faces no processing-location restrictions, and the move to a hybrid deployment can wait until a banking license, a new market or a partner bank’s due diligence introduces residency or audit demands.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.
