ServiceNow to ClickHouse: How to Move Your Data

Moving ServiceNow into ClickHouse puts your IT estate behind something fast enough to sit under a live dashboard, at a fraction of the operational overhead of a warehouse. An asset inventory somebody checks daily is a lookup, and lookups are what ClickHouse is built to answer in milliseconds.

One thing to establish immediately, because it decides whether this pipeline is what you want. The connector covers the Configuration Management Database and Software Asset Management applications. It does not carry incidents, changes, problems or requests, so if you came here for ticket analytics, the section below on scope will save you an afternoon.

ServiceNow to ClickHouse at a glance:

Why move data from ServiceNow to ClickHouse?

Two situations fit this connector well, and both are about assets rather than tickets.

The first is serving an internal tool. If ClickHouse already sits behind your operations dashboards, putting the asset inventory next to the metrics already there means one engine answering both, and an infrastructure page that loads instantly rather than calling ServiceNow on every view.

The second is joining assets to telemetry you already hold. Matching configuration items against monitoring data, logs or cost metrics is a join, and doing it in the engine that holds the heavy side avoids moving anything twice. If you want governed, long-term reporting, a warehouse suits that better.

What do you need before you start?

The first item involves a commercial conversation rather than a configuration screen:

An Airbyte enterprise agreement covering this connector. ServiceNow is a premium enterprise connector available at an additional cost, so it is not something you switch on from the catalogue. The ServiceNow source documentation sets out what it covers.

Your instance base URL and a service account. The connector authenticates with a username and password, and that account must have access to every table you want included. Its permissions define your dataset, so create one for the pipeline rather than borrowing an administrator's.

Confirmation that the streams answer your question. Check the published stream list against what you actually need before committing, because the coverage is narrower than the ServiceNow name suggests and the connector is still incubating at an early version.

A ClickHouse instance and credentials. With permission to create tables in the target database. The destination writes through the native protocol into typed columns, and at inventory volumes the smallest instance you run is ample.

If your ClickHouse instance restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a ServiceNow to ClickHouse pipeline in Airbyte?

Step 1: Match the streams to your question

Write down what somebody asked you for, then read the stream list. If the answer involves servers, network devices, printers or licence entitlements, this connector covers it. If it involves tickets, approvals or resolution times, it does not, and no configuration changes that.

Step 2: Configure the ServiceNow source

Click Sources in the left navigation, then New Source, and select ServiceNow, following adding a source. Enter your instance as the base URL, then the username and password of the service account. Run the connection test and review which streams appear.

Step 3: Configure the ClickHouse destination

Click Destinations, then New Destination, and select ClickHouse, following adding a destination. Supply host, port, database and credentials. Think about your sort order while you are here, because on a columnar engine that choice affects query performance more than anything in the pipeline does.

Step 4: Decide between a mirror and a history

Click Connections, then New connection, and choose a sync mode. Only full refresh is available, so pick overwrite if the table backs a dashboard showing the current estate, or append if you want snapshots accumulating for comparison over time.

That choice matters more here than on other destinations, because it determines whether you ever need to think about deduplication at all.

What is actually in these streams?

Two application areas. The Configuration Management Database streams describe infrastructure as configuration items: Linux, Windows, AIX and Solaris servers, ESX hosts, routers, switches, firewalls, load balancers, printers, scanners, storage switches and chassis hardware. The Software Asset Management streams cover licences, product lifecycle records and model categories.

What is absent is the half of ServiceNow most people picture. There are no incident records, no change requests, no problems and no service catalogue items, so questions about mean time to resolution, change failure rate or ticket volume by team cannot be answered from this dataset.

If those are your questions, Connector Builder against ServiceNow's Table API is the route, since the API exposes far more than this connector currently wraps. That is more work than selecting a connector and considerably less than discovering the gap after promising somebody a dashboard.

Why does full refresh make deduplication simpler here?

Because the usual ClickHouse complication does not arise. On an incremental pipeline, ReplacingMergeTree collapses duplicate rows during background merges rather than on write, so a query run shortly after a sync can return rows that the same query returns cleanly later, and exact counts need the FINAL operator.

With full refresh and overwrite there is no overlap to collapse. Each sync replaces the table with the current estate, so what you query is simply what ServiceNow last reported, and the merge timing that catches people out on other sources is not part of the picture.

Choose append and you reintroduce the question deliberately, because the same configuration item appears once per snapshot by design. Those are not duplicates to remove, so keep the snapshot date in your sort order and filter on it rather than reaching for FINAL, which would collapse exactly the history you wanted.

Frequently asked questions

Can I sync incidents and change requests?

Not with this connector. It covers Configuration Management Database and Software Asset Management streams only. For ITSM records, build against ServiceNow's Table API in Connector Builder.

Is the ServiceNow connector included in my plan?

It is a premium enterprise connector available at an additional cost, so it needs a commercial conversation rather than a configuration change.

Do I need to query with FINAL?

Not with overwrite, since each sync replaces the table and there is nothing to collapse. With append, avoid it, because repeated configuration items across snapshots are the history rather than duplicates.

Should I use overwrite or append?

Overwrite if the table backs a dashboard of the current estate. Append if you want to compare the estate over time, keeping the snapshot date in your sort order.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. You will want ClickHouse views joining configuration items to whatever telemetry you already hold.

Get your ServiceNow data into ClickHouse

Check the streams against your question before anything else, because the coverage is narrower than the name suggests. Then choose overwrite for a dashboard or append for history, keep the snapshot date in your sort order if you append, and leave FINAL alone on this pipeline.

Airbyte's connector catalog includes 700+ pre-built connectors, so asset data can sit beside the telemetry and identity sources that complete the picture. For the same source in a warehouse, see ServiceNow to Snowflake, and for cloud infrastructure activity in the same engine, AWS CloudTrail to ClickHouse.

Start syncing now →