Sendgrid to Elasticsearch: How to Move Your Data

Indexing SendGrid into Elasticsearch is how support staff answer the question customers actually ask: why did the email never arrive. That answer lives across bounces, blocks, spam reports and suppression lists, and finding it means searching by a fragment of an address rather than paging through a dashboard.

Two things decide whether this suits you. The Elasticsearch destination runs on Airbyte Core and PyAirbyte only, and SendGrid grants API key permissions per resource, so which streams you can read depends on scopes you set when creating the key.

SendGrid to Elasticsearch at a glance:

Why move data from SendGrid to Elasticsearch?

Two situations account for most of these pipelines, and both are lookups rather than reports.

The first is deliverability support. A customer says they never received a message, and the answer is somewhere across bounces, blocks, spam reports and invalid addresses. Searching all of those at once by a partial address is what an index does and what the SendGrid interface does not.

The second is putting that lookup inside your own tools, so support staff never leave the system they work in. If you want send volumes, open rates and campaign trends instead, a warehouse is the better destination because those are aggregate questions.

What do you need before you start?

One item rules the pipeline out, and one decides which streams you get:

A self-managed Airbyte deployment. The Elasticsearch destination is available on Airbyte Core and PyAirbyte, and not on the paid Cloud tiers. No configuration works around this.

A dedicated read-only API key with the right scopes. SendGrid grants permissions per resource, so the scopes on your key decide which streams work. Airbyte recommends creating a key specifically for this rather than reusing one your application already holds. The SendGrid source documentation lists which scope each stream needs.

A start date in the expected format. Supplied as a UTC date and time, with data before it not replicated. For a support lookup, going back as far as your team ever investigates is the right answer rather than the default anyone reaches for.

An index mapping you have designed. Email addresses need to work two ways at once, exactly for filtering and partially for searching, which is a specific mapping decision covered below rather than something dynamic mapping gets right.

Finally, agreement that holding customer email addresses in a search index is acceptable, with a retention period attached. Nothing expires an index for you.

How do you build a SendGrid to Elasticsearch pipeline in Airbyte?

Step 1: Create the key with every scope you need

Work out which streams answer your support question, then grant the corresponding scopes together. Full access works and is blunt; a read-only key with the specific scopes is what Airbyte recommends, and either way write down what you granted so a missing stream later is a two minute check.

Step 2: Configure the SendGrid source

Click Sources in the left navigation, then New Source, and select Sendgrid, following adding a source. Enter the API key and start date, then refresh the source schema and compare the available streams against the scopes you granted before assuming anything is missing.

Step 3: Configure the Elasticsearch destination

Click Destinations, then New Destination, and select Elasticsearch, following adding a destination. Supply your endpoint and authentication, and create the index with your mapping first, so the connector writes into a schema you designed rather than one dynamic mapping invented from the first document.

Step 4: Create the connection and schedule for support

Click Connections, then New connection, select your streams and an incremental sync mode where available. Frequency here follows how quickly support needs to see a bounce, which is usually hourly rather than daily, since the customer complaining is often complaining about this morning.

Take the suppression streams together rather than one at a time, because an address can be absent from bounces and present in blocks, and a support tool that checks only one of them gives a confidently wrong answer.

Why does the key decide which streams you get?

Because SendGrid's permission model is per resource, and the connector can only read what your key allows. Different streams require different scopes, so a key created for one purpose will quietly not cover another, and the symptom appears in the catalogue rather than as an error explaining itself.

On a support tool that failure is worse than it sounds, because a missing suppression stream does not look broken. It looks like an address that was never suppressed, which is exactly the wrong answer to give someone asking why their customer did not receive an email.

So grant the scopes deliberately, keep the record, and check every selected stream produced documents after the first sync. Using a key dedicated to this pipeline also means somebody adjusting permissions for another integration cannot silently change what your support tool can see.

How should you map an email address?

Two ways at once, because support staff need both behaviours. An exact lookup by full address has to match precisely, which is a keyword field. A search for a fragment, a surname or a domain has to match partially, which needs analysis. Choosing only one makes half your queries useless.

Elasticsearch handles this with a multi-field mapping, indexing the same value as both an analysed text field and a keyword. Declaring that yourself is the point of designing the mapping, because dynamic mapping will pick one interpretation and you will discover which when a query returns nothing.

Bounce reasons and error messages are the other field worth analysing, since they are free text from receiving mail servers and searching them by phrase is how you spot a pattern across many customers. Status codes, suppression types and timestamps stay as keyword and date fields for exact filtering.

Frequently asked questions

Why can I not find the Elasticsearch destination in Airbyte Cloud?

It is available on Airbyte Core and PyAirbyte, and not on the Standard, Plus, Pro or Enterprise Flex plans. This pipeline needs a self-managed deployment.

Why is a stream missing or empty?

Almost always a scope on the API key, since SendGrid grants permissions per resource. Add the scope, refresh the source schema, and check the stream produces documents.

Should I use a full access key?

It works and it is blunt. Airbyte recommends a dedicated read-only key, which is both safer and means another team's permission change cannot affect your pipeline.

Why does searching part of an email address return nothing?

The field is mapped as keyword only, which matches exactly. Use a multi-field mapping so the address is indexed as both analysed text and a keyword.

Can I do this without writing code?

The Airbyte setup is UI-driven. You will want to define the index mapping in Elasticsearch, which is short JSON and the part that determines whether support can find anything.

Get your SendGrid data into Elasticsearch

Check your Airbyte plan, then grant the key scopes deliberately and record them, because a missing suppression stream produces a confidently wrong answer rather than an obvious failure. Map addresses as both keyword and analysed text, take the suppression streams together, and set a retention period on the index.

Airbyte's connector catalog includes 600+ pre-built connectors, so deliverability data can serve support and reporting at once. For the aggregate view of email performance, see Customer.io to BigQuery, and for another support-facing search index, Zendesk Support to BigQuery.

Start syncing now →