Monday to Databricks: How to Move Your Data
Move Monday into Databricks with Airbyte. Why activity log retention makes a pause permanent, and how to pivot nested column values into usable tables.
Moving Monday into Databricks turns boards that teams run their work on into something you can analyse across the organisation. Monday shows one board beautifully and answers almost nothing about patterns spanning fifty of them, which is the question operations leaders actually have.
This guide covers the managed path with Airbyte. Two things shape the build: incremental sync depends on activity logs whose retention your plan decides, and board content arrives as a nested array rather than as the columns everybody sees in the interface.
Monday to Databricks at a glance:
Why move data from Monday to Databricks?
Two situations account for most of these pipelines.
The first is measuring delivery across teams. Monday is excellent at showing a team its own board and offers very little when somebody asks how long work sits in review across the whole department, or whether a process change helped. Those questions need every board in one place with a consistent shape imposed on them.
The second is joining work tracking to outcomes, so a project can be set against the revenue or the support burden that followed it. If all you want is a live view of one board, Monday's own dashboards do that better than any pipeline will, and the modelling work described below is not worth doing for a question the tool already answers.
What do you need before you start?
Four things, and the second one governs how much history you can ever hold:
A Monday API token. Generated from your account's administration area, carrying the access of whoever created it. The Monday source documentation covers generating one and the streams available.
Your plan's activity log retention, in writing. Incremental sync works from the activity logs stream rather than a timestamp, so the window those logs cover is the window your pipeline can recover from. Find the number before you design a schedule around it.
Permission to create Volumes in Unity Catalog. Staging happens through Avro files written into a Volume, and that permission is separate from being able to create tables. In organisations where Unity Catalog is administered centrally, requesting it early is sensible.
A list of the columns you actually report on. Board columns arrive as entries in a nested array rather than as real columns, so turning them into something analysts recognise is modelling work, and knowing which handful matter keeps that work small.
If your workspace restricts traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.
How do you build a Monday to Databricks pipeline in Airbyte?
Step 1: Find out how long your activity logs are kept
This number decides everything about how you operate the pipeline. Retention varies by plan, and because incremental sync reads changes from those logs, a gap longer than the retention period is a gap you cannot fill by running the sync again. Ask whoever manages your Monday account rather than assuming, and set your schedule and your alerting against that figure rather than against convenience.
Step 2: Configure the Monday source
Click Sources in the left navigation, then New Source, and select Monday, following adding a source. Supply the API token and select your streams. Include activity logs even if nobody asked for them, because that stream is what makes incremental sync possible rather than being an optional extra.
Step 3: Configure the Databricks destination
Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply the workspace details, warehouse or cluster, catalogue and schema. Nested structures need no flattening here, since Spark reads them natively, so let the column values array land exactly as it arrives.
Step 4: Create the connection and alert on failure
Click Connections, then New connection, select your streams and a sync mode. Alerting matters more here than on most pipelines, because a run of failures spanning your log retention window costs you data rather than time. Treat a failed sync as something to fix today rather than when somebody next looks.
Then build the modelling layer, because the landing tables are faithful to Monday's structure and that structure is not what anybody wants to query.
Why does pausing this pipeline lose data permanently?
Because incremental sync reads from the activity logs rather than from a timestamp on the records themselves. That is a reasonable design and it makes your pipeline dependent on something Monday expires. Once a log entry has aged out, the change it described is no longer discoverable, and no configuration recovers it.
Retention depends on your plan, which makes this a commercial question as much as a technical one. A generous plan gives comfortable margin for a broken pipeline over a holiday period. A modest one means a fortnight of failures leaves a hole in your history that no amount of re-running fixes, and the hole is invisible unless somebody happens to compare against Monday itself.
So operate accordingly. Alert on sync failure rather than reviewing occasionally, keep the schedule well inside the retention window, and if the pipeline has been down longer than that window, say so plainly rather than quietly resuming. A full refresh restores current state but not the sequence of changes in between, and anyone measuring how long work sat in a status needs that sequence.
How should Databricks hold Monday's column values?
As they arrive in bronze, then pivoted in silver. Monday does not send board columns as columns. Each item carries a nested array of entries, one per column, each holding an identifier, a display text, a type and a raw value. That is a sensible way to represent boards whose shapes differ, and it is unqueryable by anyone expecting a status column to be a status column.
Databricks is well suited to bridging that gap, because Spark handles the nested array natively and the landing table preserves the raw record beside the typed columns. Nothing is lost on the way in, which matters when somebody later asks about a column nobody modelled. The pivot then happens in a silver table that selects the handful of columns your reporting uses and gives each a proper name and type.
Two practical notes for that silver layer. Use the column identifier rather than the display text when you pivot, since somebody renaming a column in Monday should not break your model. And choose deliberately between the text and value fields, because the first is what the interface shows and the second is the underlying representation, and for statuses and dates they are not interchangeable. Write down which you used, because the next person will assume the other one.
Frequently asked questions
What happens if the pipeline is down for a month?
If that exceeds your activity log retention, the changes in the gap are gone for good. A full refresh recovers current state but not the history of how items moved through it.
Why are my board columns not columns?
Monday sends them as a nested array of entries carrying an identifier, text, type and value. Pivoting that into named columns is modelling work you do in a silver table.
Should I pivot on the column name or the identifier?
The identifier, so that renaming a column in Monday does not break your model. Names are for people and identifiers are for pipelines.
Why does the destination need to create Volumes?
Staging goes through Avro files written into a Unity Catalog Volume. That permission is separate from creating tables, and without it the pipeline cannot run.
Can I do this without writing code?
The pipeline, yes. The pivot from nested column values into something analysts can query is real modelling work, and without it the landing tables are accurate and unusable.
Get your Monday data into Databricks
Start with your activity log retention, because that number decides how much a failed sync costs and it is the one thing here you cannot engineer around. Secure Volume permission early. Let the nested column values land untouched, since Spark reads them happily, then pivot on identifiers rather than names in a silver table covering only the columns your reporting genuinely uses.
Airbyte's connector catalog includes 600+ pre-built connectors, so work tracking can be measured alongside the outcomes it produced. For the same source into a warehouse, see Monday to BigQuery, and for a comparable work management source into the same destination, Airtable to Databricks.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.