ServiceNow to Databricks: How to Move Your Data
Move ServiceNow into Databricks with Airbyte. What the CMDB and SAM streams cover, why ITSM tickets are absent, and turning full refresh into a time series.
Moving ServiceNow into Databricks puts your IT estate where it can be modelled rather than only listed. ServiceNow holds the authoritative record of what infrastructure you run and what software you are licensed for, and turning that into forecasts, anomaly detection or capacity planning needs notebooks rather than a console.
One thing to establish immediately, because it decides whether this pipeline is what you want. The connector covers the Configuration Management Database and Software Asset Management applications. It does not carry incidents, changes, problems or requests, so if you came here for ticket analytics, the section below on scope will save you an afternoon.
ServiceNow to Databricks at a glance:
Why move data from ServiceNow to Databricks?
Two situations fit this connector well, and both are about assets rather than tickets.
The first is modelling an estate that changes. Capacity forecasting, predicting which hardware is approaching end of life, and spotting configuration items that appear without a corresponding request are all questions that want Spark over a time series rather than a query over current state.
The second is joining asset data to cost. Cloud billing, procurement records and licence entitlements sit in different systems, and reconciling what you own against what you pay for is exactly the kind of multi-source work a lakehouse is built to hold.
What do you need before you start?
The first item involves a commercial conversation rather than a configuration screen:
An Airbyte enterprise agreement covering this connector. ServiceNow is a premium enterprise connector available at an additional cost, so it is not something you switch on from the catalogue. The ServiceNow source documentation sets out what it covers.
Your instance base URL and a service account. The connector authenticates with a username and password, and that account must have access to every table you want included. Its permissions define your dataset, so create one for the pipeline rather than borrowing an administrator's.
Confirmation that the streams answer your question. Check the published stream list against what you actually need before committing, because the coverage is narrower than the ServiceNow name suggests and the connector is still incubating at an early version.
Unity Catalog access. Each stream becomes a table in Unity Catalog and the connector needs permission to create Volumes for Avro staging. Agree the catalog and schema before the first sync.
If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.
How do you build a ServiceNow to Databricks pipeline in Airbyte?
Step 1: Match the streams to your question
Write down what somebody asked you for, then read the stream list. If the answer involves servers, network devices, printers or licence entitlements, this connector covers it. If it involves tickets, approvals or resolution times, it does not, and no configuration changes that.
Step 2: Configure the ServiceNow source
Click Sources in the left navigation, then New Source, and select ServiceNow, following adding a source. Enter your instance as the base URL, then the username and password of the service account. Run the connection test and review which streams appear.
Step 3: Configure the Databricks destination
Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply your workspace details, the catalog and schema, and authentication. Unlike a relational destination there is no volume ceiling to design around, which matters more here than it first appears.
Step 4: Use append and build a snapshot layer
Click Connections, then New connection, and choose a sync mode. Only full refresh is available, so pick append rather than overwrite if you want anything modelled over time, and derive a snapshot date from the extraction timestamp in your bronze layer.
Daily is generous for an inventory that changes when somebody provisions a server, and it gives you a clean daily grain to model against.
What is actually in these streams?
Two application areas. The Configuration Management Database streams describe infrastructure as configuration items: Linux, Windows, AIX and Solaris servers, ESX hosts, routers, switches, firewalls, load balancers, printers, scanners, storage switches and chassis hardware. The Software Asset Management streams cover licences, product lifecycle records and model categories.
What is absent is the half of ServiceNow most people picture. There are no incident records, no change requests, no problems and no service catalogue items, so questions about mean time to resolution, change failure rate or ticket volume by team cannot be answered from this dataset.
If those are your questions, Connector Builder against ServiceNow's Table API is the route, since the API exposes far more than this connector currently wraps. On a lakehouse that is a reasonable thing to run alongside, landing both in the same catalog so the two datasets can be modelled together.
How do you turn snapshots into a time series?
By accepting full refresh rather than fighting it. Each sync writes the complete estate as it stood that day, so appending gives you one row per configuration item per day, and that is a genuine time series even though the source offers no change feed at all.
From there the modelling is ordinary and the questions are good ones. When did this server first appear, when did it stop being reported, how long between a licence being purchased and deployed, and is the rate of new configuration items accelerating in a way that suggests somebody is provisioning outside process.
This is where the lakehouse earns its place over a relational destination. Daily snapshots of an estate multiply quickly, and a warehouse with a volume guideline would push you toward overwrite and lose the history. Here you can keep every snapshot and let a silver layer collapse them into first-seen and last-seen records.
Frequently asked questions
Can I sync incidents and change requests?
Not with this connector. It covers Configuration Management Database and Software Asset Management streams only. For ITSM records, build against ServiceNow's Table API in Connector Builder.
Is the ServiceNow connector included in my plan?
It is a premium enterprise connector available at an additional cost, so it needs a commercial conversation rather than a configuration change.
Should I use overwrite or append?
Append, in almost every case. Only full refresh is available, and appending turns each sync into a dated snapshot you can model over time. Overwrite gives you a mirror you could get from ServiceNow itself.
Will daily snapshots become too large?
Not on a lakehouse. An estate inventory is thousands of rows, so a year of daily snapshots is still small, and a silver layer can collapse them into first-seen and last-seen records.
Can I do this without writing code?
The Airbyte setup is entirely UI-driven. You will want Spark or SQL models turning the appended snapshots into a time series, which is where the analysis lives.
Get your ServiceNow data into Databricks
Check the streams against your question before anything else, because the coverage is narrower than the name suggests. Then use append rather than overwrite, derive a snapshot date in bronze, and let a silver layer collapse the snapshots into the first-seen and last-seen view that makes the history useful.
Airbyte's connector catalog includes 700+ pre-built connectors, so asset data can sit beside the identity and infrastructure sources that complete the picture. For the same source in a warehouse, see ServiceNow to Snowflake, and for cloud infrastructure activity alongside it, AWS CloudTrail to ClickHouse.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.