Move Apify Dataset data anywhere it matters.

Replicate Apify Dataset data to the warehouse or AI app you use.

Sync Apify Dataset free Read the docs

Source

Apify Dataset

Snowflake
BigQuery
Postgres
Databricks
Claude

Destinations: 50+
Standard support: 2.2.64

Overview

What is the Airbyte Apify Dataset connector?

The Airbyte Apify Dataset connector extracts data from Apify Dataset and loads it into 50+ destinations such as Snowflake, BigQuery, Databricks or Postgres, with full refresh and incremental sync. It publishes tables and events you can select per connection.

01
Sync to your warehouse
Land Apify Dataset data in 50+ destinations on a schedule you control.
02
Incremental syncs
Pull only the records that changed since the last run instead of reloading everything.
03
One authorization
Authenticate Apify Dataset once and Airbyte keeps every scheduled sync running on it.
04
Cloud or self-hosted
Run the Apify Dataset connector on Cloud, OSS, Hybrid.

Setup

Live in twelve steps

First, navigate to the Apify website and log in to your account.
Once you are logged in, click on the "API" tab in the top navigation bar.
Next, click on the "Credentials" tab and then click the "Create new token" button.
Give your token a name and select the appropriate permissions for your use case.
Copy the generated token to your clipboard.
Navigate to your Airbyte dashboard and click on the "Sources" tab.
Click on the "Add Source" button and select "Apify" from the list of available connectors.
In the "Connection Configuration" section, paste the token you copied from Apify into the "API Token" field.
Enter the name of the dataset you want to connect to in the "Dataset Name" field.
Click the "Test" button to ensure that the connection is successful.
If the test is successful, click the "Save" button to save your configuration.
You can now use the Apify source connector in Airbyte to extract data from your chosen dataset.

Configuration fields

Apify Dataset configuration fields
Field	Type	Description
dataset_id*	string	ID of the dataset you would like to load to Airbyte. In Apify Console, you can view your datasets in the Storage section under the Datasets tab after you login. See the Apify Docs for more information.
token*	string	Personal API token of your Apify account. In Apify Console, you can find your API token in the Settings section under the Integrations tab after you login. See the Apify Docs for more information.

Hybrid deployment

Sovereign by architecture, not by policy.

The data plane runs in your environment. Records, credentials and keys stay with you; the control plane only receives configuration and job metadata.

Cloud
OSS
Hybrid

Your environment

DataConnectorsCredentialsKeys

↕ metadata only

Airbyte control plane

OrchestrationMonitoringUI

FAQ

Apify Dataset connector questions

What data can I sync from Apify Dataset with Airbyte?

The Apify Dataset connector syncs the tables and events published in Airbyte docs.

Does the Apify Dataset connector support incremental sync?

Yes. The Apify Dataset connector supports incremental sync on selected streams, and full refresh on every stream.

Where can I run the Apify Dataset connector?

Run the Apify Dataset connector on Cloud, OSS, Hybrid. Hybrid deployments keep the data plane in your environment while the control plane orchestrates syncs.

Didn't find your answer? Talk to sales