Google Search Console to Databricks: How to Move Your Data

Moving Google Search Console into Databricks gives you search performance history beyond the sixteen months the interface offers, and somewhere to model it properly. Search Console answers questions about recent performance well and cannot compare this year against three years ago, because it no longer holds three years ago.

This guide covers the managed path with Airbyte. Two things shape the build: a freshness setting decides whether your figures are stable or current, and every dimension combination you want is a separate report you define rather than something you slice later.

Google Search Console to Databricks at a glance:

Why move data from Google Search Console to Databricks?

Two situations account for most of these pipelines.

The first is history. Search Console keeps a bounded window, so year-on-year comparison is impossible from the interface alone and becomes possible only if somebody started collecting earlier. A lakehouse holds that indefinitely without the volume conversation a warehouse would eventually prompt.

The second is keeping several grains at once, which is where this destination genuinely helps. Query-level data across many pages and countries is large, and the absence of a practical ceiling means you can retain the detailed reports rather than choosing one summary and discarding the rest.

What do you need before you start?

Four things, and the second shapes every figure you will ever report:

Owner or full user permission on each property. Authentication is via OAuth or a service account, and view-level access is not enough. The Google Search Console source documentation covers both routes and the custom report format.

A decision on data freshness. Final gives you only finalised, verified figures. All includes fresher data that Google has not yet settled and may revise. Neither is wrong and they produce different numbers for the same day.

Your list of site URLs and the report definitions. One source can cover several properties, which is a pleasant contrast with most connectors. Domain properties use the sc-domain prefix rather than a plain address.

Permission to create Volumes in Unity Catalog. Staging goes through Avro files written into a Volume, which is separate from creating tables and worth requesting early.

If your workspace restricts traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Google Search Console to Databricks pipeline in Airbyte?

Step 1: Choose your data freshness setting deliberately

Decide with whoever consumes this data whether they want figures that are stable or figures that are current, because you cannot have both. Choosing final means recent days are absent until Google finalises them; choosing all means recent days appear and then change. Agreeing it now avoids the conversation where somebody compares a number from today against the same number next week and assumes the pipeline is broken.

Step 2: Configure the Google Search Console source

Click Sources in the left navigation, then New Source, and select Google Search Console, following adding a source. Supply your credentials, the site URLs, a start date, the data state and any custom reports. Google's APIs Explorer is a good way to build and test a report definition before committing it here.

Step 3: Configure the Databricks destination

Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply the workspace details, warehouse or cluster, catalogue and schema. Each report becomes its own table, so name your custom reports the way you want those tables named.

Step 4: Create the connection and start collecting early

Click Connections, then New connection, select your streams and a sync mode. Daily is right. Because Search Console's own window is bounded, history beyond it exists only in what you have collected, so starting the pipeline before anybody urgently needs it is the cheapest thing you will do on this project.

If you chose the fresher setting, plan for recent days to be rewritten, since those figures are provisional by design.

Final or all, and what changes?

The data state parameter decides how settled the figures you receive are. Final restricts the query to data Google has processed, verified and considers unlikely to change, which is the definitive version and arrives with a lag. All includes fresher data that has not been through that process and may be revised afterwards.

Both are defensible and they suit different audiences. An SEO team watching whether yesterday's change helped wants the fresher view and understands it will move. A monthly report going to executives wants the finalised version, because a figure that quietly revises after being presented is worse than a figure that arrived a few days late.

If both audiences exist, run two connections rather than compromising, since the data is small enough that duplicating it is cheaper than the argument about whose number is right. Whatever you choose, record it beside the tables, because nothing in the data itself says which setting produced it and a figure's meaning depends entirely on that.

Why is each dimension combination its own report?

Because Search Console answers a query rather than handing over a dataset. You define a custom report by naming the dimensions you want, and the result becomes a stream carrying exactly that combination. There is no single wide table to slice afterwards, which makes report definition a design decision rather than a configuration detail.

Two details follow from that. Date is always included whether or not you specify it, so every report is a time series and specifying date only changes the grouping order. And the primary key is your chosen dimensions plus date, site URL and search type together, which is worth understanding before anybody writes a join expecting something simpler.

This is where the absence of a volume ceiling earns its place. Most teams on a warehouse pick one or two grains because query-level data across pages and countries is large, then discover the question they now have needs a combination they never collected. Here you can define several reports, keep them all in bronze, and build whatever aggregates the business currently wants in silver, so a new question is a modelling job rather than a request to start collecting.

Frequently asked questions

Why do recent figures keep changing?

You are probably using the fresher data state, which includes figures Google has not finalised. Switch to final for stable numbers that arrive with a lag instead.

Can one source cover several sites?

Yes, site URLs is a list. Domain properties use the sc-domain prefix rather than a plain web address, which catches people out.

Do I need to include date as a dimension?

It is included regardless. Specifying it only changes the order results are grouped in, and it forms part of the primary key alongside site URL and search type.

Why can I not see a property?

The authenticating account needs owner or full user permission on it. View-level access is not sufficient for this connector.

Can I do this without writing code?

The pipeline, yes, though report definitions are worth testing in Google's APIs Explorer first. The silver layer turning several grains into the aggregates people read is modelling work.

Get your Google Search Console data into Databricks

Settle the data freshness question first, because final and all produce different figures for the same day and nothing in the data reveals which you chose. Define your custom reports knowing each combination is a separate stream and that date is always present. Then use the room this destination gives you: keep several grains rather than picking one, and start the pipeline early, since history beyond Search Console's own window exists only in what you collected.

Airbyte's connector catalog includes 600+ pre-built connectors, so search performance can be modelled beside the traffic and revenue it produced. For the same source into an operational database, see Google Search Console to MySQL, and for commerce data into the same destination, Shopify to Databricks.

Start syncing now →