Kafka to Teradata: How to Move Your Data

Moving Kafka into Teradata brings event streams into an environment built for large-scale analytics that most organisations already run their reporting on. Topics expire; a Teradata system does not, and the questions asked there tend to span years rather than hours.

This guide covers the managed path with Airbyte. Two things shape the build: the destination does not encrypt its connection unless you tell it to, and the consumer group you choose decides what this pipeline can and cannot recover.

Kafka to Teradata at a glance:

Why move data from Kafka to Teradata?

Two situations account for most of these pipelines.

The first is bringing events to where the analysis already happens. Organisations running Teradata have years of modelled data and a reporting practice built around it, and event streams are far more valuable joined to that than sitting in a separate platform somebody has to learn.

The second is retention, since topics expire and a Teradata system is where long-lived history lives. The caveat is structural: event payloads are nested and this destination is strongly typed, so somebody has to decide what the columns are. Where payloads change often, Kafka to BigQuery absorbs that more gracefully than a strict schema will.

What do you need before you start?

Four things, and the first is a security decision rather than a connection detail:

A chosen SSL mode. Encryption is off by default and there are six modes, two of which permit an unencrypted connection. Decide which one you want before configuring, because the default is not the safe option. The Teradata destination documentation lists them.

Bootstrap servers, read permission and existing topics. The connector consumes from topics that already exist and needs an account allowed to read them. The Kafka source documentation covers the protocol and subscription settings.

A dedicated consumer group identifier. Offsets belong to the group, so reusing one that something else uses means competing for messages. Name it after this pipeline.

Teradata credentials and a logon mechanism. TD2 is the default, with LDAP and BROWSER also supported, and which one you use is usually dictated by how your organisation authenticates rather than by preference.

If your cluster or Teradata system restricts traffic by IP, add the Airbyte Cloud IP addresses to the relevant allow lists before you begin.

How do you build a Kafka to Teradata pipeline in Airbyte?

Step 1: Set the SSL mode before anything else

This is first because the default is not what most people would choose if asked. Encryption is off unless configured, and of the six available modes two will happily proceed without it, which means a configuration that looks complete can be sending your data across the network in the clear. Pick a mode that requires encryption, and have whoever owns the Teradata system confirm it rather than assuming.

Step 2: Configure the Kafka source

Click Sources in the left navigation, then New Source, and select Kafka, following adding a source. Supply the bootstrap servers, protocol, message format, subscription method and your dedicated group identifier. Use the test topic option to confirm messages are being consumed before pointing anything important at this.

Step 3: Configure the Teradata destination

Click Destinations, then New Destination, and select Teradata, following adding a destination. Supply the host, credentials, logon mechanism and SSL mode. Tables land in a default schema named airbyte_td unless you specify another, which is worth setting deliberately so the tables sit where your organisation expects rather than somewhere nobody looks.

Step 4: Create the connection and sync inside retention

Click Connections, then New connection, select your streams and a sync mode. The schedule must sit well inside your topic retention, since messages that expire before the pipeline reads them are lost rather than delayed. Alert on failure, because a quiet outage here costs data rather than freshness.

Then model the payloads into proper columns, because a strongly typed system is not a comfortable place to leave nested structures untouched.

Why does the consumer group decide what you receive?

Because Kafka tracks progress per consumer group rather than per application. The connector joins as a member of the group you name, and the offsets recorded against that group are what determine where reading resumes. Two things sharing a group identifier divide the partitions between them, so each receives part of the data and both appear to be working.

That failure is unusually hard to diagnose, because nothing errors. Your Teradata tables simply contain some of the messages, the missing ones went to whatever else shared the group, and the gap has no pattern anybody can spot from the destination. Naming the group after the pipeline avoids it entirely and costs nothing.

Retention is the related constraint. Because reading resumes from a stored offset, a pipeline paused for longer than the topic keeps messages cannot catch up; those messages are gone and the next read starts from whatever still exists. That makes topic retention the real bound on how long this pipeline may be broken, and it is worth knowing the number rather than assuming somebody set it generously.

What happens when nested payloads meet a typed system?

Somebody has to decide what the columns are. Event payloads carry nested structures because that is how services describe things to each other, and Teradata is a strongly typed relational system that expects columns with declared types. The mismatch is not a fault in either; it is simply work that has to happen somewhere.

The practical approach is to treat the landing tables as a staging layer and model deliberately above them, extracting the fields your reporting needs into typed columns with names your analysts recognise. That is familiar work in a Teradata environment, where modelling is usually a discipline rather than an afterthought, and it plays to the platform's strengths.

What it does mean is that changing payloads have a cost here that they would not have elsewhere. A producer adding a field is invisible until somebody extends the model, and a producer changing a type breaks it. Agree with the teams producing these events that payload changes are announced rather than discovered, and if that conversation is unrealistic, consider whether a destination more tolerant of drift is the better home for this particular data.

Frequently asked questions

Is the connection encrypted by default?

No. SSL is off unless you configure it, and two of the six available modes permit unencrypted connections. Choose a mode that requires encryption.

Where did my tables go?

Probably the default schema, airbyte_td, which is used unless you specify another. Set it deliberately so tables land where your organisation expects.

Some messages never arrive.

Check whether anything else uses the same consumer group, since partitions are divided between members and each receives only part of the data without anything erroring.

Can the pipeline catch up after a long outage?

Only within your topic retention. Messages that expired while the pipeline was down are gone, so retention is the real limit on how long it may be broken.

Can I do this without writing code?

The pipeline, yes. Modelling nested payloads into typed columns is real work, and in a Teradata environment it is the work that makes the data usable.

Get your Kafka data into Teradata

Set the SSL mode first, because encryption is off by default and two modes will proceed without it. Give the pipeline its own consumer group, since sharing one splits the data silently between members. Sync well inside your topic retention and alert on failure, as expired messages cannot be recovered. Then model the payloads into typed columns, and agree with producing teams that payload changes get announced rather than discovered.

Airbyte's connector catalog includes 600+ pre-built connectors, so event streams can reach the systems your analysts already use. For the same source into an operational database, see Kafka to PostgreSQL, and for a relational source into the same destination, Oracle Database to Teradata.

Start syncing now →