ServiceNow to Amazon Redshift: How to Move Your Data
Move ServiceNow into Amazon Redshift with Airbyte. What the CMDB and SAM streams cover, why ITSM tickets are absent, and the mandatory S3 staging bucket.
Moving ServiceNow into Amazon Redshift puts your IT estate next to the AWS spend it largely explains. If your analytical estate already runs on AWS, the asset inventory and the cost and usage reports can finally sit in one place rather than being reconciled by hand each quarter.
One thing to establish immediately, because it decides whether this pipeline is what you want. The connector covers the Configuration Management Database and Software Asset Management applications. It does not carry incidents, changes, problems or requests, so if you came here for ticket analytics, the section below on scope will save you an afternoon.
ServiceNow to Amazon Redshift at a glance:
Why move data from ServiceNow to Amazon Redshift?
Two situations fit this connector well, and both are about assets rather than tickets.
The first is reconciling assets against AWS cost. Cost and usage reports land in S3 and are commonly queried from Redshift, so putting the CMDB beside them lets you ask which configuration items account for which spend without exporting either side to a spreadsheet.
The second is infrastructure history. The CMDB describes your estate as it is now, and a warehouse accumulating dated snapshots tells you how it changed, which servers appeared during a migration and what quietly persisted after a decommissioning project was declared finished.
What do you need before you start?
One commercial conversation, one credential, and an AWS prerequisite:
An Airbyte enterprise agreement covering this connector. ServiceNow is a premium enterprise connector available at an additional cost, so it is not something you switch on from the catalogue. The ServiceNow source documentation sets out what it covers.
Your instance base URL and a service account. The connector authenticates with a username and password, and that account must have access to every table you want included. Its permissions define your dataset, so create one for the pipeline rather than borrowing an administrator's.
An S3 staging bucket and IAM permissions. The Redshift destination writes data to S3 as files with a manifest, then issues a COPY command, which is Redshift's own recommended approach. You need put, get, delete and list on the bucket, in the same region as the cluster.
Redshift entities created in advance. A database, a schema and a user able to write data and manage staging operations. Airbyte writes into two schemas, the one you nominate and a raw schema it manages, so tell whoever reviews your Redshift objects that the second is expected.
Note also that an SSH tunnel covers the SQL connection only. Staging traffic reaches S3 over HTTPS regardless, so a network design assuming everything passes through a bastion is wrong.
How do you build a ServiceNow to Amazon Redshift pipeline in Airbyte?
Step 1: Match the streams to your question
Write down what somebody asked you for, then read the stream list. If the answer involves servers, network devices, printers or licence entitlements, this connector covers it. If it involves tickets, approvals or resolution times, it does not, and no configuration changes that.
Step 2: Configure the ServiceNow source
Click Sources in the left navigation, then New Source, and select ServiceNow, following adding a source. Enter your instance as the base URL, then the username and password of the service account. Run the connection test and review which streams appear.
Step 3: Configure the Redshift destination
Click Destinations, then New Destination, and select Redshift, following adding a destination. Supply the cluster host, database, schema and credentials, along with your S3 bucket and IAM key. Staging files are removed after a successful COPY by default, and can be retained if you need them for debugging.
Step 4: Choose append and set your keys
Click Connections, then New connection, and choose a sync mode. Only full refresh is available, so pick overwrite for a mirror or append to accumulate snapshots. If you append, set the sort key on the extraction timestamp, because every historical query will filter on it.
Daily is generous for an inventory that changes when somebody provisions a server, and weekly is defensible if you only compare month to month.
What is actually in these streams?
Two application areas. The Configuration Management Database streams describe infrastructure as configuration items: Linux, Windows, AIX and Solaris servers, ESX hosts, routers, switches, firewalls, load balancers, printers, scanners, storage switches and chassis hardware. The Software Asset Management streams cover licences, product lifecycle records and model categories.
What is absent is the half of ServiceNow most people picture. There are no incident records, no change requests, no problems and no service catalogue items, so questions about mean time to resolution, change failure rate or ticket volume by team cannot be answered from this dataset.
If those are your questions, Connector Builder against ServiceNow's Table API is the route, since the API exposes far more than this connector currently wraps. That is more work than selecting a connector and considerably less than discovering the gap after promising somebody a dashboard.
Why does a SaaS source need an S3 bucket?
Because the requirement comes from the destination rather than the source. Redshift is designed to be loaded in bulk from S3, so the connector writes files with a manifest and then issues a COPY command pointing at them. That is Redshift's own recommended path and there is no direct insert alternative.
On this pipeline the surprise is milder than most, because the people reconciling ServiceNow against AWS cost reports are already working in an account with S3 buckets and IAM roles. The bucket is a planning item rather than a new capability to acquire.
Keep it in the same region as your cluster to avoid cross-region transfer charges, and remember that inventory volumes make this a correctness question rather than a cost one. A few thousand configuration items passing through staging will not register on any bill.
Frequently asked questions
Can I sync incidents and change requests?
Not with this connector. It covers Configuration Management Database and Software Asset Management streams only. For ITSM records, build against ServiceNow's Table API in Connector Builder.
Is the ServiceNow connector included in my plan?
It is a premium enterprise connector available at an additional cost, so it needs a commercial conversation rather than a configuration change.
Do I need an S3 bucket to load into Redshift?
Yes. The destination stages data in S3 and loads it with COPY, which is Redshift's recommended approach. Put the bucket in the same region as your cluster.
Why is there an extra schema in my cluster?
Airbyte writes to your target schema and to a raw data schema it manages to improve reliability. Both are expected.
Can I do this without writing code?
Yes. The setup above is entirely UI-driven. You will want Redshift views joining configuration items to your cost and usage data, which is where the reconciliation lives.
Get your ServiceNow data into Amazon Redshift
Check the streams against your question before anything else, because the coverage is narrower than the name suggests. Set up the staging bucket in the same region as your cluster, choose append so you accumulate an estate history, and sort on the extraction timestamp since that is what every historical query filters on.
Airbyte's connector catalog includes 700+ pre-built connectors, so asset data can sit beside the cost and infrastructure sources that explain it. For the same source in a warehouse without staging, see ServiceNow to Snowflake, and for cloud audit activity alongside it, AWS CloudTrail to Redshift.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.