Instagram to BigQuery: How to Move Your Data

Moving Instagram into BigQuery lets you connect organic social performance to everything else: which posts brought people who signed up, whether follower growth tracks revenue, how content performs against paid campaigns. Instagram's own insights answer none of that, because they end at engagement.

One scope note first, because it saves a wasted project: this connector covers organic content. For performance data about Instagram Ads, the Facebook Marketing source is the one you want. This guide covers the managed path with Airbyte, and a scheduling tension between two of the connector's behaviours.

Instagram to BigQuery at a glance:

Why move data from Instagram to BigQuery?

Two situations account for most of these pipelines.

The first is connecting organic content to outcomes. Instagram reports reach and engagement, and your own systems know who signed up or bought. Neither answers whether the content is doing anything commercially, and a warehouse holding both does.

The second is history and comparison. Instagram's interface is built around recent performance, and questions about how engagement has changed across two years, or how organic compares with paid on the same terms, need a long window and your own definitions applied consistently.

What do you need before you start?

The Meta side takes longer to arrange than the BigQuery side, so start it first:

An Instagram business account connected to a Facebook page. Along with a Meta for Developers account and the Instagram Graph API added to your Facebook app. A personal Instagram account will not work, and converting one is a change somebody who runs the account needs to make. The Instagram source documentation covers the prerequisites.

A decision about whether you need stories. This one shapes your whole schedule, because the Instagram API only returns stories that are live at the time of the sync. Expired stories are not available, so a story you did not capture is gone.

A deliberate start date. Entered with a date and time. If the field is blank all data is replicated, which on a long-running account is a large first sync against an API with hourly rate limits.

A BigQuery service account and dataset. With the BigQuery User and BigQuery Data Editor roles. Set the dataset location to match your paid media and revenue datasets, since comparison is the point and BigQuery only joins datasets in the same location.

One expectation to set with whoever asked for this: metrics available from Instagram change as Meta revises its API, and metrics deprecated on their side stop appearing. Treat the metric list as something to check occasionally rather than a fixed contract.

How do you build an Instagram to BigQuery pipeline in Airbyte?

Step 1: Decide whether stories are in scope

Ask whether anyone will analyse stories. If not, this is a simple daily pipeline. If so, you are committing to syncing several times a day forever, because a story lives for a day and only live ones are retrievable. Making that call now avoids designing a schedule twice.

Step 2: Configure the Instagram source

Click Sources in the left navigation, then New Source, and select Instagram, following adding a source. Authenticate your Instagram account through the Meta flow and set a start date. The connector reads through the Instagram Graph API, so a failure here is usually a missing app permission or an account that is not a business account.

Step 3: Configure the BigQuery destination

Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project, dataset and service account key. Batched standard inserts is the default and is ample, because social metrics are small by warehouse standards even across several accounts and years.

Step 4: Split stories into their own connection

Click Connections, then New connection, and select streams with a sync mode. If stories are in scope, put them and story insights in a connection of their own running several times a day, and leave users, media and their insights on a daily connection. That separation is the resolution to the tension described below.

Instagram limits requests per hour, and the connector retries automatically with exponential backoff when it hits those limits. So a frequent stories connection is workable, provided the streams it carries are the small ones.

Why do stories force a different schedule?

Because the Instagram API only returns stories that are live at the time of the sync. Expired stories are not available at all, which means there is no backfill and no catching up: a story that expired before your sync ran never existed as far as your warehouse is concerned.

A story lasts about a day, so a daily sync captures some of them and misses others depending on timing. Several syncs a day is the only way to be reasonably complete, and even then a story posted and deleted between runs is lost. That is a property of Instagram rather than of the connector, and no configuration changes it.

So tell whoever asked for stories analysis that the data begins when the pipeline does and will have gaps in proportion to how often it runs. If that is unacceptable, capturing stories properly is a job for something that reacts to publication rather than a scheduled sync.

Why not just sync everything frequently?

Because only one stream syncs incrementally. User Insights uses a date cursor and reads what is new; everything else, including media and media insights, is re-read on every sync. So the frequency stories demand is exactly the frequency that makes the rest expensive.

On a long-established account with thousands of posts, re-reading every media object and its insights six times a day is a great deal of work against an API with hourly limits, to produce data that changed marginally. It is also unnecessary, because engagement on an old post moves slowly.

Hence two connections. Stories and story insights run often and are small, since only live stories exist at any moment. Media, users and the insight streams run daily, where a full re-read is affordable. It is more objects to manage and it is the difference between a pipeline that works and one that spends its day re-reading 2019.

Frequently asked questions

Can I get historical Instagram stories?

No. The API returns only stories that are live when the sync runs, and expired stories are unavailable. Sync several times a day if stories matter, and expect gaps.

Does this include Instagram Ads performance?

No. This connector covers organic users, media and stories with their insights. For paid performance data, use the Facebook Marketing source.

Which streams sync incrementally?

Only User Insights, which uses the date field as its cursor. Everything else is re-read on each sync, which is why a frequent schedule is expensive for those streams.

Why do different posts have different metrics?

The metrics requested vary by media type, so reels, videos, carousel albums and images return different sets. Account for that when modelling rather than expecting one uniform shape.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. You will want BigQuery views normalising the differing metric sets by media type and joining performance to your own outcomes.

Get your Instagram data into BigQuery

Confirm the business account setup first, decide whether stories are in scope because that sets your whole schedule, split them into their own frequent connection while everything else runs daily, and tell your marketing team that story history begins when the pipeline does.

Airbyte's connector catalog includes 600+ pre-built connectors, so organic social can sit beside the paid media and revenue data that give it meaning. For the paid side, see Facebook Ads to Snowflake, and for lifecycle marketing on the same audience, Braze to BigQuery.

Start syncing now →