Polygon Stock API to Kafka: How to Move Your Data

Moving Polygon Stock API into Kafka publishes market aggregates where several systems can consume them at once. A pricing service, a risk model and a reporting warehouse all want the same bars, and pointing each at the API separately means three sets of credentials and three implementations of the same request.

This guide covers the managed path with Airbyte. Two things shape the build, and both catch people out: a source covers exactly one ticker, and prices you have already published can change afterwards because of how splits are handled.

Polygon Stock API to Kafka at a glance:

Why move data from Polygon Stock API to Kafka?

Two situations account for most of these pipelines.

The first is fan-out of a shared reference series. Several systems need the same daily or hourly bars, and publishing once means one caller against the API and one definition of what a bar is, rather than three teams quietly disagreeing about whether prices are adjusted.

The second is feeding services rather than analysts. Be clear that this is not a trading feed: syncs are scheduled batches, so latency is your interval and anything needing live prices needs a streaming market data provider instead. If the goal is analysing price history rather than distributing it, Exchange Rates API to ClickHouse shows the shape that suits a column store better.

What do you need before you start?

Four things, and the first determines how large this project really is:

A list of tickers, and an honest count. The ticker is a required configuration field and a source covers one of them, so a watchlist becomes a source and a connection per symbol. The Polygon Stock API source documentation lists every configuration field.

An agreed bar size. Multiplier and timespan together define the aggregate, so one day, five minutes or one hour is a decision baked into the source rather than something consumers choose later.

A decision about split adjustment. Results are adjusted for splits by default, and setting this to false gives you unadjusted prices. Neither is wrong and they answer different questions, which is exactly why it should be a decision.

Topics created in advance. The destination writes to topics that already exist. Decide whether every ticker shares one topic, which most consumers prefer, or each gets its own.

If your organisation restricts access by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Polygon Stock API to Kafka pipeline in Airbyte?

Step 1: Settle the bar size and the adjustment setting

Both are baked into the source and both define what consumers receive. Agree the multiplier and timespan with whoever uses the data, because changing them later means every consumer's assumptions change at once. Agree the adjustment setting with the same people, since adjusted and unadjusted prices answer different questions and a consumer expecting one and receiving the other will produce confidently wrong numbers.

Step 2: Configure the Polygon Stock API source

Click Sources in the left navigation, then New Source, and select Polygon Stock API, following adding a source. Supply the API key, ticker, multiplier, timespan and start date. The limit defaults to five thousand base aggregates and accepts up to fifty thousand, which matters when backfilling fine-grained bars over a long window.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and topic configuration. Messages are JSON, each wrapping the bar alongside its identifier, the extraction timestamp and the stream name, so consumers read through an envelope.

Step 4: Create the connections and label the contract

Click Connections, then New connection for each ticker, selecting the stream and a sync mode. Write down the bar size and adjustment setting beside the topic name, because nothing in the messages themselves tells a consumer whether these prices are adjusted, and that is not something anybody should have to guess about.

Match the schedule to the bar size. Daily bars need a daily sync, and polling every minute for a daily aggregate achieves nothing but requests.

What does one source actually publish?

Aggregate bars for one ticker, at a size you fixed when you configured it. That is three constraints in one sentence and each surprises somebody. The ticker is singular, so a portfolio means many pipelines. The bars are aggregates rather than individual trades. And the bar size is configuration rather than something a consumer can vary.

The multiplication is the practical problem. A watchlist of fifty symbols is fifty sources and fifty connections, all identical except for one field, and all capable of failing independently. Naming them consistently matters, and so does having a single place that records which symbols are covered, because a missing ticker is invisible from the topic.

Be equally clear about latency. This polls on your schedule rather than streaming, so the bus carries market data without being a market data feed. Anything making decisions in seconds needs a different architecture entirely, and the fact that the data arrives on Kafka makes it look more real-time than it is, which is precisely the misunderstanding worth heading off before consumers are built.

Why do prices you already published change?

Because results are adjusted for splits by default, and a split rewrites history. When a company splits its stock, every prior price in an adjusted series is restated to keep the series comparable, so bars you published last month legitimately carry different values when they are read again. Nothing has gone wrong; the definition of the series has been applied to a new event.

On a bus that is more awkward than in a table. A warehouse can simply update the affected rows, whereas a topic accumulates both the old and new versions of the same bar, and compaction cannot help because messages are keyed by a generated identifier rather than by ticker and timestamp. A consumer reading naively will hold two prices for the same minute and no rule for choosing.

So give consumers a rule. Key on ticker and bar timestamp, take the version with the latest extraction timestamp from the envelope, and treat earlier ones as superseded. If several consumers need that, do it once in a stream processor and publish the resolved series to a second topic. And if restatement is unwelcome, setting adjusted to false gives a series that never changes retrospectively, at the cost of prices that are not comparable across a split.

Frequently asked questions

Can one source cover several tickers?

No. The ticker is a single required field, so each symbol needs its own source and connection. Name them consistently and keep a record of which are covered.

Is this a real-time market feed?

No. Syncs are scheduled batches, so latency is your interval. Arriving on Kafka makes it look live, and anything trading on these prices needs a streaming provider instead.

Why did an old bar's price change?

A split occurred and the series is adjusted by default, so prior prices were restated. Set adjusted to false if you want a series that never changes retrospectively.

Can consumers tell whether prices are adjusted?

Not from the messages, which is why the setting belongs in documentation beside the topic name along with the bar size.

Can I do this without writing code?

The pipelines, yes. The consumers are yours, and each needs a rule for choosing between versions of the same bar after a restatement.

Get your Polygon Stock API data into Kafka

Count your tickers honestly, because each is a source and a connection. Agree the bar size and the adjustment setting before anybody builds a consumer, and record both beside the topic since the messages do not carry them. Tell consuming teams this is polled rather than streamed. And give them a rule for restatement: key on ticker and bar timestamp, prefer the latest extraction, because splits rewrite history and compaction will not tidy it.

Airbyte's connector catalog includes 600+ pre-built connectors, so reference data can reach every system that depends on it. For periodic analytics data onto the same bus, see Amplitude to Kafka, and for a database feeding the same bus, PostgreSQL to Kafka.

Start syncing now →