Google Search Console to BigQuery: How to Move Your Data
Move Google Search Console into BigQuery with Airbyte. Why a rough pipeline today beats a perfect one next quarter, and how to keep the archive cheap.
Moving Google Search Console into BigQuery builds a search performance archive that outlives Google's own retention. The interface reaches back about sixteen months, so anything older exists only in what you collected, which makes the start date of this pipeline the most consequential thing about it.
This guide covers the managed path with Airbyte. Two things shape the build: the value compounds the earlier you begin, and search data is granular enough that query cost becomes a design consideration rather than an afterthought.
Google Search Console to BigQuery at a glance:
Why move data from Google Search Console to BigQuery?
Two situations account for most of these pipelines.
The first is retention. Sixteen months covers this year against last and not much else, so questions about a redesign two years ago or a seasonal pattern across three years are unanswerable unless somebody started collecting.
The second is joining search performance to conversion and revenue, which Search Console has never seen. If interactive speed across that archive matters more than joins, Google Search Console to ClickHouse answers those aggregations faster.
What do you need before you start?
Four things, and the first is often a request to somebody else:
Owner or Full User access to each property. Restricted access is not enough to read the API. The Google Search Console source documentation covers authentication and the report options.
Your site URLs in the right form. The field takes a list, and a domain property is written with the sc-domain prefix rather than as an ordinary address.
A BigQuery dataset in the right location. Location is fixed at creation and BigQuery will not join across locations, so put this where your conversion and revenue data already live.
A view on how granular your reports should be. Each dimension combination becomes its own stream, and the most detailed ones produce enormous tables that cost money to query.
If your network restricts traffic by IP, add the Airbyte Cloud IP addresses to the relevant allow list before you begin.
How do you build a Google Search Console to BigQuery pipeline in Airbyte?
Step 1: Start collecting before the modelling is agreed
Get a basic pipeline running as soon as you have access, even if nobody has settled which reports they want, because every week of delay is a week that will never be recoverable. Search Console keeps about sixteen months and gives none of it back. Refining dimension sets later is easy; retrieving a month you failed to collect is impossible.
Step 2: Configure the Google Search Console source
Click Sources in the left navigation, then New Source, and select Google Search Console, following adding a source. Authenticate, list your site URLs and set a start date and data state. One source can cover several properties, so most businesses need only one.
Step 3: Configure the BigQuery destination
Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project identifier, dataset and service account credentials. A query-level report on a busy site is one of the cases where Cloud Storage staging is worth considering.
Step 4: Create the connection and build the views first
Click Connections, then New connection, select your streams and a sync mode. Daily is right for search data. Then write the views people will query, with a filter on the partitioning column built in, before anybody starts writing their own.
Record which data state you chose beside the tables, since verified and fresh figures differ for recent days and nothing in the data says which you took.
Why does starting early matter so much?
Because this is one of the few pipelines where delay costs something irreplaceable. Search Console holds roughly sixteen months, so the archive you can build is bounded at the front by that window and at the back by whenever you began. Every other decision here can be revised; this one cannot.
The value compounds in a way that is easy to underestimate at the start. In year one you have what Search Console already shows you and the pipeline looks redundant. In year three you can compare a redesign against the two years before it, which is exactly the question nobody could answer in year one and precisely why somebody eventually asks for this.
So treat a rough pipeline running now as better than a considered one running next quarter. Take a sensible default set of reports, get access sorted, and refine the dimension sets once people have opinions. The tables you add later start their history the day you add them, which is a perfectly acceptable outcome as long as the core ones started earlier.
How do you keep an archive cheap to query?
By filtering the partition and being disciplined about dimensions, because search data is granular and BigQuery bills on bytes scanned. Tables arrive partitioned on the extraction timestamp, and a query ignoring that column reads every day you have ever collected, which is the whole point of the archive and an expensive way to answer a question about last week.
Dimension choice is the other half. A report broken down by query, page, country and device produces a row for every combination that saw traffic, most of which had a handful of impressions, and that table grows faster than anything else you will create here. The detail is occasionally invaluable and routinely ignored.
So build the views before the habits form. A view per common question, each filtering the partitioning column and selecting only the dimensions that question needs, means analysts inherit the right pattern rather than discovering it after a surprising bill. Keep the granular report if somebody genuinely needs it, and make sure nothing queries it by default.
Frequently asked questions
How far back can I backfill?
About sixteen months, which is Search Console's own window. Anything older was never available, so start collecting before you think you need to.
Why are my queries expensive?
Probably not filtering on the extraction timestamp partitioning column, so every query scans the full archive. Build that filter into your views.
Should I request every dimension?
No. Each combination is its own stream, and the most granular reports are enormous and mostly low-traffic rows. Request what your questions need.
The connection fails on a property.
Check the access level, since Owner or Full User is required, and that domain properties use the sc-domain prefix rather than a normal URL.
Can I do this without writing code?
The pipeline, yes. The views that filter the partition and shape each question are SQL, and they are what keeps the archive affordable.
Get your Google Search Console data into BigQuery
Start collecting as soon as access allows, because sixteen months is all Google keeps and a rough pipeline running now beats a considered one next quarter. Confirm Owner or Full User access and write domain properties with the sc-domain prefix. Then build views that filter the partitioning column and request only the dimensions your questions need, before anybody forms expensive habits.
Airbyte's connector catalog includes 600+ pre-built connectors, so search performance can outlive Google's own retention. For the same source into a lakehouse, see Google Search Console to Databricks, and for the same source into an operational database, Google Search Console to MySQL.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.