How to load data from Omnisend to Firebolt

Begin by logging into your Omnisend account. Navigate to the section where your data is stored—typically under ‘Reports’ or ‘Contacts’. Use the available export function to download your data in a CSV or Excel format. Ensure that you choose the appropriate data range and include all necessary fields for your analysis or storage needs.

Once you have your data exported, open the file in Excel or a similar spreadsheet application. Verify that all the necessary fields are included and clean the data by removing any duplicates or irrelevant columns. Ensure that the data types are consistent (e.g., dates, numbers, text) to prevent issues during the import process.

Firebolt requires data to be structured in specific formats. Convert your CSV or Excel file into a format compatible with Firebolt, such as Parquet or JSON. Use a tool or script to perform this conversion, ensuring that the structure is maintained and data types are preserved. Parquet is often preferred for its efficiency in storage and query performance.

Log into your Firebolt account. If you haven’t already, set up a database and table schema that matches the structure of your data. Ensure that your table schema aligns with the data types and format of your exported data. Define the primary keys, indexes, and any partitioning strategies to optimize performance.

Use Firebolt’s built-in data loading capabilities to transfer your formatted data file. You can do this via the Firebolt SQL command interface. Use the `COPY INTO` command to load your data into the Firebolt table. For example:
```sql
COPY INTO my_table
FROM 's3://your-bucket/data-file.parquet'
CREDENTIALS=(aws_key_id='AWS_KEY' aws_secret_key='AWS_SECRET');
```
Ensure the data source path and credentials are correctly specified.

After the data has been loaded, perform a set of queries to verify that the data has been transferred correctly. Check for completeness, ensure that all records are present, and validate the integrity by comparing a few records against the original dataset. Look for any inconsistencies or discrepancies.

Once data is successfully loaded and verified, optimize your table for better query performance. Update your table indexes and partitions if needed, based on typical query patterns. Use Firebolt’s optimization features such as aggregating indexes and partitioning strategies to ensure efficient data retrieval and storage performance.

By following these steps, you can efficiently move data from Omnisend to Firebolt without relying on third-party connectors or integrations.