Create Streams Using Any XML-based Endpoint with Connector Builder

Create custom streams from XML-based APIs with Airbyte Connector Builder and simplify data ingestion from flexible endpoint sources.

ENGINEERING

October 21, 2024

5 min read

Quinton Wall

Summarize with AI:

Create Streams Using Any XML-based Endpoint with Connector Builder

If you caught any of the 1.0 release news, you will have seen a big focus on the long tail of connectors and how important this is to powering data and AI pipelines. In order to support as many endpoints as possible we recently updated the Connector Builder to support application/xml content types.

‍

Why is this a big deal?

‍

In a perfect world, all endpoints would support JSON payloads, but we are not there yet, and probably never will be. Whilst JSON dominates modern APIs, primarily due to performance of sending smaller payloads across the wire, readability of these payloads, and most importantly, native support within JavaScript, XML remains extremely relevant in legacy systems and systems that require very complex structures. Now, with support for application/xml, we’ve unlocked all of these endpoints for you to create connectors using no code. Here’s how it works.

‍

Let’s say I want to use the OpenWeather API to retrieve 5 day forecast weather data for San Francisco. Yup, OpenWeather supports both JSON and XML, but bear with me. For our example, we are going to use the XML option. Regardless of which format you want, the first thing I need to do in Connector Builder is set up my base URL and authentication via a query parameter, appId.

‍

Next, set up the stream to use the forecast endpoint. Knowing that our response format is going to be XML, all I need to do is choose XML in the HTTP Response Format selector. Airbyte will take care of the rest and map data to JSON records.

‍

Once I have my response format, looking at the openweather API docs, I need to pass in the lat and long of the location as query parameters. Since I want XML format, I need to add an additional parameter, mode, and set it to xml. With everything configured, it’s time to test the stream.

‍

Everything looks like it is working and retrieving data. You can see the raw XML response returned from OpenWeather and how Airbyte automatically translates this to JSON formatted records. Upon closer inspection though, we need to tweak the records response to correctly map to the data we need. This is no different to any other stream when working with Connector Builder. Thankfully though, I don’t have to try and navigate a complicated XML schema, responses are always in JSON. This makes it very easy to add the right field path values to get the records we want. In this example, weatherdata, forecast, and time.

‍

Testing the stream again, we retrieve the results we want. But come to think of it, we are talking about San Francisco here. It’s probably going to be foggy every day ;)

‍

That’s it. With just a quick selection of the new XML HTTP Response Format, Connector Builder is now capable of working with any XML-based API. If you want to learn more, check out the online docs and join our slack community to share your feedback.

‍

Integrate with 700+ apps using Airbyte

Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.

Try it free Talk to sales

About the Author

Quinton Wall

Quinton is the Head of Developer Relations at Airbyte

Integrate with 700+ apps using Airbyte

Try Airbyte for free

Create Streams Using Any XML-based Endpoint with Connector Builder

Integrate with 700+ apps using Airbyte

Integrate with 700+ apps using Airbyte

Related posts