Parquet File to Teradata: How to Load Your Data

Moving Parquet files into Teradata is usually about getting data that arrived as files into the analytical platform the rest of the business queries. Partner extracts, exports from another system, output from a Spark job: all of it lands as files, and none of it is useful to a reporting tool until it is in a table.

One thing to clear up first, because it sends people looking for a connector that does not exist. There is no standalone Parquet source. Parquet is a file format, so you choose the file-based source matching where your files live and tell it to read them as Parquet. This guide covers that path, and a Teradata default that deserves a moment's attention.

Parquet File to Teradata at a glance:

Why move Parquet files into Teradata?

Two situations account for most of these pipelines.

The first is making a file feed queryable. Data sitting in object storage is invisible to everybody until somebody loads it, and if your reporting layer is built on Teradata then that is where it needs to be. A scheduled load turns a directory nobody can use into a table anybody can join against.

The second is consolidation. Teradata already holds your enterprise data, and file extracts from other systems need to sit beside it rather than in a separate analytical stack. If your querying happens in a lakehouse instead, leaving the files where they are and registering them as tables is cheaper than moving them.

What do you need before you start?

The first item is a choice rather than a credential, and the rest follow from it:

The right file-based source for where your files live. S3, Azure Blob Storage, Google Cloud Storage, SFTP and others each have their own connector, and every one of them can read Parquet. Searching the catalogue for Parquet finds nothing useful; searching for your storage does.

Credentials for that storage, and a glob pattern. The pattern selects which files belong to the stream, and being specific matters, because a loose pattern that catches two differently shaped feeds produces one confused table.

Confidence that your files agree with each other. Parquet carries its own schema, which is mostly a gift and becomes a problem when files in the same stream declare different ones. Ask whoever produces them whether the schema has changed over time.

Teradata credentials and a logon mechanism. A host and a user able to create tables and write data. TD2 is the default logon mechanism, with LDAP and BROWSER also supported, and which one you use is usually already decided by whoever administers the Vantage instance.

If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list on the Teradata side before you begin.

How do you build a Parquet to Teradata pipeline in Airbyte?

Step 1: Check the files agree before you load them

Open a recent file and an old one and compare their schemas. Parquet declares its types, so this takes minutes, and it is the check that prevents the most common failure on this pipeline: a stream whose files disagree about what a column contains. If they have drifted, split them into separate streams rather than hoping.

Step 2: Configure the file-based source

Click Sources in the left navigation, then New Source, and select the connector for your storage, following adding a source. Supply the storage credentials, set the file format to Parquet, and enter your glob pattern. The connector reads the schema from the files rather than sampling them, which is the main advantage of this format over CSV.

Step 3: Configure the Teradata destination

Click Destinations, then New Destination, and select Teradata, following adding a destination. Supply the host, credentials and logon mechanism, then set the schema name rather than accepting the default of airbyte_td, because on a Vantage instance with established conventions an unexpected schema appearing is the sort of thing that generates a ticket.

Step 4: Turn on SSL, then create the connection

Encryption on the Teradata destination is off by default. Enable it and choose a mode, then click Connections, New connection, and select your stream with a sync mode. Incremental works from file creation or modification time, so new files are picked up without re-reading the ones already loaded.

Size the first run deliberately. Parquet compresses well, so a directory that looks modest on disk can expand considerably once loaded as rows, and a first sync across years of files is a different proposition from a nightly increment.

Why is Parquet's embedded schema both a gift and a constraint?

The gift is that nothing has to be guessed. A Parquet file declares its columns and their types, so a number arrives as a number and a timestamp as a timestamp, without the sampling and inference that CSV and JSON sources depend on. That removes a whole category of problem, including the sparse-field misses that affect schemaless sources.

The constraint is that the schema is a fact rather than a suggestion. If the job producing your files changed a column's type last spring, you now have two populations of file that genuinely disagree, and a stream reading both has to reconcile something that cannot be reconciled. Nothing about the format smooths that over.

So the practical work is in the glob pattern. Where a feed has changed shape, partition the files by that boundary and load them as separate streams, then reconcile in Teradata where you have SQL and can decide explicitly how the old shape maps to the new. That is more honest than hoping one stream absorbs both.

Why is the encryption default worth pausing on?

Because SSL is off unless you enable it, and nothing warns you at setup time. You turn it on explicitly and then choose from six modes, ranging from disabling encryption entirely through to verifying the server certificate fully.

Two of those modes deserve particular care. Prefer and allow both permit an unencrypted connection under some conditions, which is rarely what anyone intends when they go to the trouble of enabling SSL at all. If data crosses a network you do not control, choose require at minimum, and one of the verifying modes if you have the certificates.

It is worth a line in your design document rather than a box somebody ticks, because the data heading into a Teradata instance is usually the data an organisation is most careful about, and a default that quietly allows plaintext is exactly the kind of thing an audit finds later.

Frequently asked questions

Where is the Parquet source connector?

There is not one, because Parquet is a file format rather than a system. Choose the connector for wherever your files live and set the format to Parquet.

Is my Teradata connection encrypted by default?

No. SSL is off unless you enable it and select a mode. Use require or one of the verifying modes for anything crossing a network you do not control.

What happens if my files have different schemas?

You get trouble, because Parquet declares its types rather than inferring them. Split the differing populations into separate streams with your glob pattern and reconcile them in Teradata.

Why did tables appear in an unfamiliar schema?

The destination writes to a default schema when none is specified. Set it explicitly to match your Vantage conventions.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. You may want SQL in Teradata to reconcile files whose schema changed over time, which is where that decision belongs.

Get your Parquet data into Teradata

Pick the source by where your files live rather than by format, compare an old file against a new one before loading anything, set the schema name to something your DBA recognises, and turn SSL on with a mode that actually encrypts. Then size the first run knowing Parquet compresses well.

Airbyte's connector catalog includes 600+ pre-built connectors, so file feeds can reach a warehouse and a lake at once. For the same format landing in a key-value store, see Parquet File to DynamoDB, and for a database feeding the same destination, Oracle Database to Teradata.

Start syncing now →