Background
Data Stack
Data sources: Smartsheets, MSSQL, Facebook, GitHub, Google Sheets, etc.
Integration (EL): Airbyte
Orchestration: Prefect
Transformation: dbt
Warehouse: Snowflake
Data Quality: Great Expectations, Monte Carlo
For more than a century, Graniterock has provided the California construction industry with quality materials and products for building the perfect infrastructure, whether a major highway, airport runway, or bridge.
Over the last few decades, as data and cloud technology matured, the Graniterock team relied on an increasingly complex infrastructure stack to get the most out of their ever-increasing data.
Graniterock leverages its data warehouse for a wide variety of applications. From resource scheduling and job forecasting for the Construction Division, COVID tracking for the HR and Safety Divisions, material production for the Sales and Operational Divisions, and even customer engagement for the Marketing Division.
The Challenge
But the Graniterock team faced a challenge. Their existing tools were expensive, difficult to maintain and coordinate, and prone to data integrity issues.
Additionally, since the team uses many industry-specific data endpoints that require custom connectors, they spent lengthy development hours on connector development and maintenance.
The Solution
Enter Airbyte, the leading open-source data integration platform, and Prefect, the new standard in workflow automation.
Together, these tools presented a framework for creating and automating new connectors. This allowed the Graniterock team to standardize their data processes using a common language—and build out scalable pipelines with much-needed visibility in the flow of data.
Using Airbyte, engineers no longer need to spend hours on connector development, as the company’s pipelines are now built on out-of-the-box connectors with customizable features.
The addition of Prefect enables the Graniterock team to schedule and orchestrate their entire data pipeline: kicking-off an Airbyte sync, triggering the right dbt transformation, and on completion, beginning Great Expectations testing. Further, Prefect provides detailed insights into the pipeline metrics useful for troubleshooting and performance improvements within the flow.
Results
Now, the Graniterock team moves at the speed their business requires, without the hold up of lengthy development processes.
“The combination of Airbyte and Prefect reduced internal development time and expenses for Graniterock by more than 50%, getting the data in the hands of analysts faster than ever before.” - Cody Kaiser, Enterprise Data Manager
Faster time to insight allowed the team to build complicated dashboards to manage daily operational metrics and quickly troubleshoot data errors with minimal downstream interruptions. As an added bonus, their overall costs for data tools decreased by 25%.
“Both Prefect and Airbyte have amazing staff who have helped us custom develop solutions that are specific to our industry and business. They take time to listen, understand the problem and provide meaningful solutions and support.” - Cody Kaiser, Enterprise Data Manager
