RSS to Kafka: How to Move Your Data

Moving RSS into Kafka publishes feed items where several systems can react to them. A search index, an alerting service and an archive all want the same announcements, and pointing each at the feed means three pollers doing identical work against somebody else's server.

This guide covers the managed path with Airbyte. Two things shape the build: one source covers exactly one feed, so monitoring many publications multiplies quickly, and feed items have no identifier the pipeline can rely on.

Rss to Kafka at a glance:

Why move data from Rss to Kafka?

Two situations account for most of these pipelines.

The first is fan-out to systems that treat announcements differently. One consumer indexes them for search, another matches them against watchlists and raises alerts, a third keeps an archive for compliance. Publishing once is politer to the publisher's server than three independent pollers and simpler for you to operate.

The second is normalising many feeds into one stream that consumers read without caring where an item came from. With a single consumer, though, a bus is overhead and a direct pipeline into a database is simpler to build and easier to query afterwards.

What do you need before you start?

Four things, and the first one determines how much work this actually is:

A list of the feeds you want, and a realistic count. A source takes one feed URL, so each publication is its own source and connection. Ten feeds is ten of everything, which is tedious rather than difficult but worth knowing before you promise a delivery date.

Confirmation that each feed publishes dates. Incremental sync uses the published date as its cursor, so a feed that omits or mangles them behaves unpredictably. The RSS source documentation sets out what the connector expects.

Topics, created before the first sync. The destination writes to topics that already exist. Decide whether every feed shares one topic or each gets its own, since that choice shapes what consumers have to filter.

Consumers that expect to see items more than once. Feed items have no reliable identifier, so repeats reach the topic and nothing upstream can prevent them. That has to be designed for rather than patched later.

If your organisation restricts access by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build an Rss to Kafka pipeline in Airbyte?

Step 1: Decide how feeds map onto topics

One topic for everything means consumers filter by source and get a single stream to subscribe to. A topic per feed means consumers choose what they care about and you maintain more topics. For a handful of feeds the first is usually right, and for feeds with very different audiences, such as regulatory notices against industry press, separating them saves every consumer from filtering. Settle it now, because changing it later means rewriting consumers.

Step 2: Configure the RSS source

Click Sources in the left navigation, then New Source, and select RSS, following adding a source. Supply the feed URL, which is the only field. Repeat for each feed, and name the sources after the publication rather than the URL so the list stays readable once there are fifteen of them.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and topic configuration. Messages are JSON, and each wraps the item alongside its identifier, the extraction timestamp and the stream name, so consumers read through an envelope rather than finding the feed's own shape at the top level.

Step 4: Create the connections and be honest about latency

Click Connections, then New connection for each feed, selecting the stream and a sync mode. This is polling on a schedule rather than a push, so an item reaches consumers at your sync interval rather than at publication. Hourly suits most feeds, and anything faster is more requests against a publisher's server for very little gain.

Be considerate about frequency generally, since these are usually somebody else's servers and nothing obliges them to keep serving you.

What does one source actually cover?

One feed, and that is the whole configuration. There is no list of URLs and no pattern matching, so a project described as monitoring the industry press becomes as many sources and connections as there are publications. Each is trivial to set up and the multiplication is the point worth planning for.

That changes how you think about adding feeds later. Somebody asking to include three more publications is asking for three more sources, three more connections and three more things that can quietly stop working. Naming conventions matter more than they would on a single pipeline, and so does having somewhere that records which feeds are covered.

It also makes monitoring the pipelines a real task rather than an afterthought. A feed that changes its URL, moves behind a redirect or simply stops publishing will fail or go quiet, and with twenty connections nobody notices one of them idling. Alert on failure, and consider a simple check that flags any feed which has produced nothing for longer than it usually goes between items.

Why can't consumers trust they have seen an item once?

Because feed items have no guaranteed identifier. The RSS specification makes guid optional, so the connector offers no primary key and works from a cursor on the published date instead. When a publisher edits an item, republishes it or adjusts a timestamp, the pipeline has no way to recognise it as something already sent.

Kafka offers no help either. Log compaction keeps the latest message per key, and messages here are keyed by an Airbyte-generated identifier rather than anything from the item, so every copy carries a different key and there is nothing for compaction to collapse. Repeats persist for the full retention period however the topic is configured.

So deduplication belongs in the consumers, keyed on the item's link, which is effectively unique in most feeds even when guid is absent. A consumer raising alerts particularly needs this, since sending the same notification twice is exactly the failure that erodes trust in an alerting system. If several consumers need it, an enrichment step that deduplicates once and republishes to a second topic saves each of them implementing the same logic slightly differently.

Frequently asked questions

Can one source cover several feeds?

No, a source takes one feed URL. Each publication needs its own source and connection, which is straightforward but multiplies what you maintain.

Do items reach consumers as they are published?

No. This polls the feed on your schedule, so latency is the sync interval. Hourly is reasonable and anything faster mostly adds load to somebody else's server.

Will compaction remove repeated items?

No, because messages are keyed by a generated identifier rather than by the item. Consumers should deduplicate on the item's link instead.

Should each feed have its own topic?

It depends on whether consumers care about the difference. One shared topic is simpler for a handful of similar feeds; separate topics suit feeds with genuinely different audiences.

Can I do this without writing code?

The pipelines, yes, and each is a one-field setup. The consumers are yours, and every one of them needs deduplication logic since the pipeline cannot provide it.

Get your Rss data into Kafka

Count your feeds honestly, because each one is a source and a connection rather than a line in a list. Decide early whether they share a topic or get their own, since consumers depend on that contract. Then tell consuming teams that items will repeat, that compaction will not help because messages are keyed by a generated identifier, and that deduplicating on the link is the practical answer, particularly for anything sending alerts.

Airbyte's connector catalog includes 600+ pre-built connectors, so public publications can reach every system that watches them. For team conversation onto the same bus, see Slack to Kafka, and for build activity onto the same bus, Jenkins to Kafka.

Start syncing now →