Harness the power of Zendesk Support data with a seamless data stack. Airbyte, Dbt, BigQuery, and Dagster come together to enable Customer Satisfaction Analytics.
Download our free guide and discover the best approach for your needs, whether it's building your ELT solution in-house or opting for Airbyte Open Source or Airbyte Cloud.
Welcome to the "Customer Satisfaction Analytics Stack" repository! ✨ This is your go-to place to easily set up a data stack using Zendesk Support, Airbyte, Dbt, BigQuery, and Dagster. With this setup, you can pull Zendesk Support data, extract it using Airbyte, put it into BigQuery, and play around with it using dbt and Dagster.
This Quickstart is all about making things easy, getting you started quickly and showing you how smoothly all these tools can work together!
Below is a visual representation of how data flows through our integrated tools in this Quickstart. This comes from Dagster's global asset lineage view:
Before you embark on this integration, ensure you have the following set up and ready:
Get the project up and running on your local machine by following these steps:
For Linux & Mac:
For Windows:
How to create a dataset:
How to create a service account and assign roles:
How to generate JSON key:
Airbyte allows you to create connectors for sources and destinations, facilitating data synchronization between various platforms. In this project, we're harnessing the power of Terraform to automate the creation of these connectors and the connections between them. Here's how you can set this up:
Change to the relevant directory containing the Terraform configuration for Airbyte:
Within the infra/airbyte directory, you'll find three crucial Terraform files:
Adjust the configurations in these files to suit your project's needs. Specifically, provide credentials for your BigQuery connection. You can utilize the variables.tf file to manage these credentials.
This step prepares Terraform to create the resources defined in your configuration files.
Before applying any changes, review the plan to understand what Terraform will do.
After reviewing and confirming the plan, apply the Terraform configurations to create the necessary Airbyte resources.
Once Terraform completes its tasks, navigate to the Airbyte UI. Here, you should see your source and destination connectors, as well as the connection between them, set up and ready to go.
dbt (data build tool) allows you to transform your data by writing, documenting, and executing SQL workflows. Setting up the dbt project requires specifying connection details for your data platform, in this case, BigQuery. Here’s a step-by-step guide to help you set this up:
Change to the directory containing the dbt configuration:
You'll find a profiles.yml file within the directory. This file contains configurations for dbt to connect with your data platform. Update this file with your BigQuery connection details.
To keep your credentials secure, you can leverage environment variables. An example is provided within the profiles.yml file.
Once you’ve updated the connection details, you can test the connection to your BigQuery instance using:
If everything is set up correctly, this command should report a successful connection to BigQuery.
If you would like to run the dbt models manually at this point, you can do so by executing:
You can verify the data has been transformed by going to BigQuery and checking the transformed_data dataset.
Dagster is a modern data orchestrator designed to help you build, test, and monitor your data workflows. In this section, we'll walk you through setting up Dagster to oversee both the Airbyte and dbt workflows:
Switch to the directory containing the Dagster orchestration configurations:
Dagster requires certain environment variables to be set to interact with other tools like dbt and Airbyte. Set the following variables:
Note: The AIRBYTE_PASSWORD is set to password as a default for local Airbyte instances. If you've changed this during your Airbyte setup, ensure you use the appropriate password here.
With the environment variables in place, kick-start the Dagster UI:
Open your browser and navigate to: http://127.0.0.1:3000. Here, you should see assets for both Airbyte and dbt. To get an overview of how these assets interrelate, click on "view global asset lineage". This will give you a clear picture of the data lineage, visualizing how data flows between the tools.
In the Dagster UI, click on "Materialize all". This should trigger the full pipeline. First the Airbyte sync to extract data from Faker and load it into BigQuery, and then dbt to transform the raw data, materializing the staging and marts models.
Congratulations on deploying and running the Customer Satisfaction Analytics Quistart! 🎉 Here are some suggestions on what you can explore next to dive deeper and get more out of your project:
Download our free guide and discover the best approach for your needs, whether it's building your ELT solution in-house or opting for Airbyte Open Source or Airbyte Cloud.