Apify to BigQuery: How to Move Your Data

Moving Apify into BigQuery turns the output of a scraping job into something you can query alongside everything else. An Apify Actor produces a dataset, and that dataset is perfectly readable through the console or an export, neither of which helps when you want to compare this week's results against last quarter's.

This guide covers the managed path with Airbyte. Two things shape the build: a source is configured against one dataset identifier rather than an Actor, and the general-purpose stream carries everything under a single key rather than as columns.

Apify to BigQuery at a glance:

Why move data from Apify to BigQuery?

Two situations account for most of these pipelines.

The first is turning scraping into monitoring. A competitor price check or a listings crawl is interesting once and valuable as a series, and that only happens if each run's output accumulates somewhere queryable rather than sitting in a dataset nobody revisits.

The second is joining scraped data to your own, such as setting external prices against your margins. The honest caveat is that a genuinely one-off scrape does not need a pipeline, since exporting the dataset directly is quicker than configuring a connector you will then have to maintain. Build this when the crawl recurs.

What do you need before you start?

Four things, and the second is the one that shapes the architecture:

An API token and a dataset identifier. The token is in the Apify Console under Settings, then Integrations, and the dataset identifier under Storage, then Datasets. That is the entire configuration. The Apify Dataset source documentation covers both.

A plan for how datasets map to syncs. Each Actor run produces its own dataset with its own identifier, so a recurring crawl generates a new one every time while the source points at exactly one. How you reconcile that is the central design question here.

A BigQuery dataset in the right location. Location is fixed at creation and BigQuery will not join across locations, so put this where the data you intend to compare against already lives.

Tolerance for an alpha connector. This is community supported at an early release stage, so the interface and available streams may change. Fine for internal monitoring, worth a conversation before something customer-facing depends on it.

If your network restricts traffic by IP, add the Airbyte Cloud IP addresses to the relevant allow list before you begin.

How do you build an Apify to BigQuery pipeline in Airbyte?

Step 1: Decide how a recurring crawl reaches one dataset identifier

A source points at one dataset, and a scheduled Actor produces a new dataset per run, so the two do not line up on their own. The tidy answer is to have each run append into a single named dataset that the pipeline reads, or to use Apify's webhook to trigger a sync when a run finishes. Settle this before configuring anything, because the alternative is somebody updating a dataset identifier by hand every morning.

Step 2: Configure the Apify Dataset source

Click Sources in the left navigation, then New Source, and select Apify Dataset, following adding a source. Supply the token and dataset identifier. If your data came from the Website Content Crawler Actor, the dedicated stream for it carries a static schema, which is considerably more pleasant to work with than the general-purpose one.

Step 3: Configure the BigQuery destination

Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project identifier, dataset and service account credentials. Scraped datasets are usually modest by warehouse standards, so batched standard inserts are ample and staging is unnecessary.

Step 4: Create the connection and consider triggering it from Apify

Click Connections, then New connection, select your stream and a sync mode. A fixed schedule works and fits awkwardly with crawls of unpredictable duration. Apify can call the Airbyte API when an Actor run finishes, which syncs exactly when there is something new rather than hoping the crawl completed before the schedule fired.

Rate limits are unlikely to trouble you, since the connector uses Apify's own client and handles the platform's limitations under normal usage.

What exactly does a dataset identifier point at?

One storage object, not an Actor and not a schedule. Apify stores the results of a job in a dataset, and a recurring Actor creates a fresh dataset each time it runs. The connector reads the identifier you gave it, which means a source configured against yesterday's run will keep reading yesterday's results indefinitely and report success every time.

That is the most common misunderstanding with this connector and it fails silently rather than loudly. The pipeline runs, rows arrive, nothing errors, and the data simply never changes. Somebody notices weeks later when a chart has been flat for a suspiciously long time.

Two arrangements avoid it. Configure your Actor to write into one persistent named dataset that accumulates across runs, and point the pipeline at that. Or keep per-run datasets and use Apify's webhook to trigger a sync as each run finishes, which suits crawls whose duration varies. Either works; what does not work is a fixed identifier and a scheduled sync, which is the combination people reach for first.

What arrives in BigQuery when the schema is dynamic?

A column holding everything. The general-purpose stream uses a dynamic schema so it can accept output from any Actor, and it achieves that by placing all the scraped fields under a single data key. That is what makes the connector universal, and it means BigQuery receives a nested structure rather than the tidy columns your scraper's output suggested.

BigQuery handles that perfectly well and nobody enjoys querying it directly. The right pattern is a view extracting the fields you care about into properly named and typed columns, built once, with everything downstream reading from it. If your data comes from the Website Content Crawler Actor, the dedicated stream provides a static schema instead and saves you this work entirely, which is worth checking before you write any extraction logic.

Scraped data also deserves defensive handling in that view, because a website changing its markup produces missing or oddly shaped fields without anything failing. Cast deliberately, expect nulls, and consider counting them as a quality signal, since a sudden rise in empty fields usually means the site changed rather than the business did. Tables partition on the extraction timestamp, so filter on it as well as any date inside the scraped data or queries will read everything you have ever collected.

Frequently asked questions

Why is my data never changing?

The source points at one dataset identifier, and a recurring Actor creates a new dataset per run. Either write each run into one persistent dataset, or trigger syncs from an Apify webhook.

Why is everything in one column?

The general-purpose stream uses a dynamic schema with all fields under a data key, which is how it supports output from any Actor. Extract what you need in a view.

Is there a stream with real columns?

Yes, for datasets produced by the Website Content Crawler Actor, which has a dedicated stream with a static schema. It only supports that Actor's output.

Can a finished crawl trigger the sync?

Yes. An Apify webhook can call the Airbyte API when an Actor run completes, which suits crawls whose duration varies and avoids syncing when nothing is new.

Can I do this without writing code?

The pipeline, yes. Extracting fields from the dynamic structure is SQL, and the webhook arrangement is configuration in Apify rather than in Airbyte.

Get your Apify data into BigQuery

Work out how a recurring crawl reaches a single dataset identifier before anything else, because a fixed identifier with a scheduled sync produces a pipeline that succeeds forever and never changes. Use the Website Content Crawler stream if it applies, since a static schema saves real work. Otherwise extract fields into a view, cast defensively because websites change without warning, and filter on the partitioning column to keep queries cheap.

Airbyte's connector catalog includes 600+ pre-built connectors, so scraped data can be compared against your own without a bespoke loader. For a general API source into the same destination, see Public API to BigQuery, and for another loosely structured source into the same destination, MongoDB to BigQuery.

Start syncing now →