Parquet File to Teradata: How to Load Your Data

Loading Parquet files into Teradata Vantage joins two things that were designed decades apart, and most of the friction lives in that gap. Parquet is a columnar file format carrying its own schema, built for object storage and distributed compute. Teradata is a mature analytical database with strong typing, its own authentication mechanisms and its own opinions about how data arrives.

One clarification before setup, because it decides where you start. There is no connector called Parquet. Parquet is a file type you select inside a file-based source, so the question is where the files live: an S3 bucket, an SFTP server, Azure Blob Storage, or somewhere similar. Pick the source connector that matches the storage, then choose Parquet as the format. Everything after that is common to all of them.

Parquet's schema is an asset, and a constraint

Unlike CSV, a Parquet file knows what its columns are and what types they hold. That removes the guesswork that makes CSV ingestion tedious, and it introduces a different problem: Parquet supports nested structures, and Teradata's relational model does not have a natural home for them.

If your files came out of Spark or a data lake pipeline, they very likely contain structs and repeated fields. Decide where those get flattened. Doing it upstream, by writing a flattened Parquet variant specifically for this pipeline, is usually cleaner than trying to reason about how a nested field is represented once it reaches a Teradata column. It also gives you an obvious place to drop the columns nobody loads, which on wide analytical Parquet is often most of them.

Schema evolution deserves the same forethought. Parquet files written months apart routinely differ, because a column was added or a type widened. A file-based source discovers the schema from what it reads, so a directory containing several generations of a dataset can produce a catalogue that surprises you. Keep generations in separate paths, or accept that you will refresh the schema whenever the writer changes.

Teradata configuration, and the one insecure default

The destination needs a host, a user with permission to create tables and write data, and a logon mechanism. TD2 is the default, with LDAP and BROWSER also supported, and which one you use is usually already decided by whoever administers the Vantage instance.

The setting worth pausing on is SSL, because encryption is off by default. You enable it explicitly and then choose a mode, and the six available modes range from disabling encryption entirely through to verifying the server certificate fully. Prefer and allow both permit an unencrypted connection under some conditions, which is rarely what anyone actually intends. If you are moving data across a network you do not control, choose require at minimum and one of the verifying modes if you have the certificates. Nothing warns you about this at setup time.

The default schema is named airbyte_td unless you change it, and it is used whenever the source does not specify a namespace. On a Vantage instance with established conventions, an unexpected schema appearing is the kind of thing that generates a ticket. Set it to something your DBA expects. Airbyte adds its own metadata columns to each table alongside your data, including a generation identifier that increments each time you run a refresh, which is useful for telling reload attempts apart.

Sizing the sync

Parquet compresses well, which is exactly why it misleads people about volume. A 500 MB file is not 500 MB of rows, and a directory that looks modest in object storage can be a substantial load against a Teradata instance sized for something else. Check the row counts, not the file sizes, before scheduling anything frequent.

Whether you can sync incrementally depends on the source connector rather than on Parquet. File-based sources that track file creation or modification time will pick up only new files; ones that do not will re-read the directory. Confirm which behaviour you have before pointing a connection at a bucket that grows daily.

Frequently asked questions

Which connector reads Parquet files?

Whichever file-based source matches where your files live, with Parquet chosen as the file type. There is no standalone Parquet connector.

Is my Teradata connection encrypted by default?

No. SSL is off unless you enable it and select a mode. Use require or one of the verifying modes for anything crossing a network you do not control.

How are nested Parquet fields handled?

Teradata has no natural equivalent for structs and repeated fields. Flatten them upstream by writing a flattened Parquet variant for this pipeline rather than resolving it at the destination.

Why did new tables appear in an unfamiliar schema?

The destination writes to a default schema when the source does not specify a namespace. Set it explicitly to match your Vantage conventions.

Get your Parquet data into Teradata

Choose the source connector by where the files live, flatten nested fields before they reach the pipeline, turn SSL on, and set the schema name to something your DBA recognises. The Parquet side is the easy half; the decisions that matter are about types and about a default that is quietly permissive.

For the same file format going somewhere with a very different type model, see Parquet to Elasticsearch. For loading an established analytical warehouse from another warehouse rather than from files, see BigQuery to Snowflake.

Start syncing now →