Agentic Data Engineering Resources

Resource

NDCG Explained: How to Measure Ranking Quality in Retrieval Systems

How NDCG formulas, metric tradeoffs, and production monitoring work together to evaluate retrieval and RAG pipeline ranking quality.

Pedro Lopez

July 28, 2026

Summarize with AI:

NDCG (Normalized Discounted Cumulative Gain) measures how well a retrieval system ranks results by relevance, rewarding systems that put highly relevant items near the top and discounting items buried lower down. It matters for retrieval and agent pipelines because the model only sees what the retriever ranks first, so the order itself is a quality signal.

The sections ahead cover the math, the two formula variants that quietly disagree, how to pick NDCG over other metrics, and how to monitor it in production.

TL;DR

NDCG measures ranking quality by rewarding highly relevant results that appear near the top of a retrieved list.
Linear and exponential NDCG formulas can produce different scores because they weight relevance grades differently.
Use NDCG when order and graded relevance matter, and pair it with Recall@K to check whether relevant items are retrieved at all.
Reliable NDCG monitoring depends on graded qrels, a K that matches production behavior, and slicing scores by query intent or source.

Try Airbyte Agents to see how consistent, typed context makes ranking metrics like NDCG measurable across your enterprise sources.

What Is NDCG and How Does It Measure Ranking Quality?

NDCG is a score between 0 and 1 that tells you how close a ranked list of results is to the ideal ordering for a query. A score of 1 means the system produced the best possible order, and lower scores indicate that relevant results appear further down than they should.

The metric builds up in three steps:

1. Cumulative gain ignores position

Cumulative gain sums the relevance scores of all retrieved items. It is order-blind: shuffling the same set of results does not change the value, which makes it useless for measuring ranking on its own. Because cumulative gain treats a relevant result at position 1 the same as one at position 9, it cannot tell a good ranking from a bad one. You need a way to make position count.

2. The log discount rewards results near the top

Discounted cumulative gain (DCG) fixes this by dividing each result's gain by the log of its position, so lower ranks contribute less. The intuition is that the gap between positions 1 and 2 matters more than the gap between positions 9 and 10, because users and models pay far more attention to the top of a list.

The standard form divides by log₂(i+1), and the +1 offset exists because log₂(1) is 0, and you cannot divide by zero. Järvelin and Kekäläinen introduced this discounting in their 2002 paper, choosing the log specifically because it reduces value as rank increases without dropping too steeply.

3. Normalization against the ideal makes scores comparable

DCG alone is not comparable across queries, because a query with many highly relevant documents can produce a much larger raw DCG than a query with few. Normalization solves this.

You compute the ideal DCG (IDCG) by sorting all relevant items into their best possible order, then dividing the actual DCG by IDCG. The result falls between 0 and 1, and because every score is relative to that query's ideal, NDCG values average cleanly across the entire query set. This is the cross-query comparability that makes NDCG useful for evaluating a retriever feeding a RAG pipeline or an agent, where the ranked output is what gets fed to the model.

Why Do Two NDCG Formulas Give Different Numbers?

Two DCG formulas are common in ranking evaluation. The first is the linear form, rel_i / log₂(i+1), where gain is proportional to the relevance grade. The second is the exponential form, (2^rel_i − 1) / log₂(i+1), which amplifies the difference between high and low grades, so a very relevant result pulls far more weight than a somewhat relevant one.

This causes a common, concrete headache. A developer computes DCG by hand with the linear formula, then compares it with an implementation, benchmark, or textbook formula that uses the exponential form, gets different numbers, and assumes something is broken. Nothing is broken. The two formulas simply weight relevance grades differently.

The two formulas diverge in how they weight relevance grades, which is why a hand calculation and a library call can disagree.

Property	Linear variant	Exponential variant
Formula	rel_i / log₂(i+1)	(2^rel_i − 1) / log₂(i+1)
Effect of relevance grade	Proportional to grade	Amplifies high grades over low grades
Example source	scikit-learn linear option	Stanford IIR textbook
Best for	Simple graded scales where steps are evenly spaced	Scales where a highly relevant result should dominate a merely relevant one
Practitioner risk	Diverges from exponential implementations	A larger spread between grades can surprise teams expecting linear behavior

With grades on a 0 to 3 scale, the linear form treats a grade-3 result as worth three times a grade-1 result. The exponential form treats it as worth 2^3 − 1 = 7, compared with 2^1 − 1 = 1, a sevenfold gap. On scales where you want a top-relevance result to dominate, the exponential form fits better; on evenly spaced scales, the linear form is fine.

The related gotcha is edge-case handling. Libraries and evaluators may differ on ties, cutoff handling, and the way they handle unusual inputs. Before you trust a number, check which variant and which edge-case behavior the library uses. For example, scikit-learn warns that ignore_ties=True gives wrong results if ties exist, and that negative values in y_true can push output outside [0, 1].

When Should You Use NDCG Instead of MRR, MAP, or Recall@K?

The right ranking metric depends on how the downstream system actually consumes the results. A generator that reads ten chunks, an agent that only looks at the first tool response, and a recall-focused sanity check each need something different, so the choice of metric follows the shape of the consumer.

Metric	Rank-aware	Relevance type	Use it when
Hit Rate@K	No	Binary (per query)	Debugging early, checking whether any relevant item appears in the top K
Recall@K	No	Binary	Confirming the relevant chunks are even present in the top K
MRR	Yes (first hit)	Binary	The generator or agent only reads the first result, common in tool routing
MAP@K	Yes (linear decay)	Binary	Many relevant items matter and relevance is treated as all-or-nothing
NDCG@K	Yes (log decay)	Graded	Order matters and relevance comes in degrees, the default for retrieval and reranking

NDCG is the only metric in this group that accepts graded relevance, which is why it became the default for search and reranking evaluation. In practice, pair it with Recall@K so you know whether the relevant chunks were even retrieved before you argue about the order they came back in. And remember that per-call NDCG stays blind to agent trajectories, so single-query metrics belong alongside end-to-end checks, never instead of them.

What Does a Low or Misleading NDCG Score Tell You?

NDCG = 1 means the system produced the ideal order up to K. NDCG = 0 means nothing relevant appeared in the top K. Everything in between is a relative position relative to that query's ideal. The number itself is easy to read; the traps sit in how the score is produced and aggregated.

Binary labels hide graded failures. A retriever can score high on NDCG computed against binary labels while quietly placing partially relevant content at rank 1, because binary labels treat a partial match as fully relevant. The fix is to relabel the eval set on a graded scale and compute NDCG against those grades. The metric may be right while the labels are wrong.
Cutoff-aware normalization means NDCG@5 can legitimately exceed NDCG@3. Each cutoff uses a different normalization, so a higher score at a larger K is not a bug.
NDCG does not penalize extra bad documents on its own. Unless your relevance grading or retrieval size accounts for them, junk further down the list will not show up in the score.
Aggregates hide segment-level regressions. A single mean can look flat while one query intent, one query length, or one source has collapsed underneath it.

Slice your scores before you act on them, and remember that low numbers often trace back to upstream data quality, which is why agent-ready data matters as much as the metric itself. A pre-materialized retrieval layer, like a Context Store, keeps the inputs to the ranker stable so the score reflects the retriever and doesn't shift payloads underneath it.

How Do You Compute and Monitor NDCG in Production?

The outcome you want is a reproducible NDCG number you can track over time and use to gate releases.

Relevance judgments (qrels) are the foundation

Qrels map query IDs to document IDs to graded relevance, in the form {query_id: {doc_id: grade}}. Use a graded relevance scale for the grades. Build a labeled query set from production traces, benchmark data, or hand labeling, with at least one judgment per query. The judgments are where the metric earns its trust, so this is the part worth investing in.

Match K to what the system actually reads

K should match the number of results the downstream system actually reads, which is why common choices cluster around the values that production retrievers feed to the model. The normalization takes k into account, so the cutoff is part of the score definition rather than a display setting.

NDCG@5 fits when the generator reads a small top-k of chunks. It is sensitive to the very top of the ranking, so small reorderings near rank 1 move the score noticeably.
NDCG@10 is a common benchmark and production cutoff. It fits systems that pass roughly 10 retrieved items downstream and is a practical default when your system returns that many results.
NDCG@20 is better suited to wider context windows or recall-leaning pipelines, at the cost of diluting the influence of the top positions.

Unless your production top-k differs, start at NDCG@10.

Wire NDCG into CI and drift monitoring

For in-process scoring, scikit-learn's ndcg_score takes y_true, y_score, and k, returning a float in [0., 1.]. Mind its warnings: ignore_ties=True gives wrong results if ties exist, and negative values in y_true can push output outside [0, 1]. For qrel and run files, TREC-style evaluators like pytrec_eval and ranx handle the same job and integrate with benchmark datasets. Scripting retrieval calls through an Agent SDK or replaying them from an Agent CLI makes it easier to keep the CI loop reproducible against the same queries over time.

The CI loop is straightforward. Run the retriever in CI, capture top-k document IDs per query, compute a metric vector (NDCG@k, Recall@k, MRR), store both per-query and aggregate, then fire a drift alert when NDCG@10 regresses on production traffic. Catching ranking drift early shortens the diagnostic chain. These patterns apply whether your retrieval uses pre-indexed or just-in-time strategies, as covered in dynamic context retrieval.

The honest limit: when production systems lack ground-truth labels, teams fall back to per-chunk relevance checks or LLM-as-a-judge scoring, which approximate ranking quality rather than replace NDCG.

What Changes When an Agent Ranks Results Across Many Enterprise Sources?

A production agent often retrieves and ranks across heterogeneous systems at once, such as a CRM, a support desk, and a billing system. Relevance means something different in each, so build qrels per source rather than grading a support ticket and an invoice with the same rubric.

One aggregate score across mixed sources hides whether one source's ranking collapsed while others held steady. Break the number down by query intent, query length, and source to identify which integration regressed.

Runtime data stitching is often the underlying cause. Live API calls can hit rate limits, return partial responses, or fail after expired OAuth breaks auth. Stale indexes can rank old records above current ones, missing permissions can hide the true best result, and cross-source reasoning can push too much context into the model until token-window pressure crowds out the evidence the answer needs. For B2B teams, those failures show up as failed agent responses, slower retriever releases, and token waste from passing poorly ranked context downstream.

How Does Airbyte Agents Support Reliable Retrieval Ranking?

Airbyte Agents is the context layer for AI agents, built to keep the inputs to any ranking-quality measurement stable across sources. Because the agent ranks over consistent, typed records rather than stitching fragmented payloads at runtime, per-source evaluation and drift tracking become practical rather than guesswork.

Context Store as a unified retrieval layer. The Context Store pre-materializes operational data from connected SaaS sources into a single searchable layer, so the agent ranks over typed records with a consistent schema rather than raw API payloads.
Two execution modes for the same query. Search runs against the indexed replica for fast reads, and Direct falls back to the live API when freshness demands it, so retrieval quality does not depend on which mode a given call needs.
Programmatic control for evaluation harnesses. The Agent SDK lets you script retrieval calls inside a CI loop, and the Agent CLI makes it straightforward to replay queries, capture top-k results, and score them against qrels.
Observable reads by default. Every retrieval is logged with its source and mode, which is what makes per-source NDCG and drift alerts meaningful rather than aggregate guesses.

These properties turn NDCG from a number you compute in a notebook into a metric you can actually monitor across a CRM, a support desk, and a billing system without the underlying inputs shifting under you.

Ready to Make Your Agent's Ranking Measurable?

NDCG is only as trustworthy as the judgments and inputs behind it. Honest graded labels, a K that matches how the system actually reads results, per-source slicing, and end-to-end checks alongside per-call scores are what turn the metric from a number you compute into a number you can act on.

The binary-labels trap catches teams that trusted a high score with weak labels; the aggregation trap catches teams that never sliced by source or intent. Treat NDCG as a number you can distrust rather than just compute.

Stable inputs make stable measurements, and that is where Airbyte Agents fit. The Context Store gives agents consistent, typed records to rank over, and the Agent SDK and Agent CLI let you wire that retrieval into CI and drift monitoring.

Get a demo today.

Frequently Asked Questions

Can you compute NDCG without graded relevance labels?

Yes, NDCG works with binary labels (1 relevant, 0 not), but it then loses its main advantage over MRR and MAP, since it cannot distinguish a highly relevant result from a merely relevant one. For implicit feedback like clicks, NDCG can reward guessing the test-set order rather than true relevance unless that order is itself relevance-graded.

Why did my worst-case ranking still score above zero?

A fully reversed short list does not always score 0 because even a poor order still yields some discounted gain relative to the ideal. NDCG hits 0 only when no relevant items appear in the top K, so a nonzero floor for a bad ranking is expected, not a bug.

What NDCG difference between two systems is meaningful?

A raw gap can be noise. Run a significance test across your full query set rather than trusting a single aggregate, and confirm the difference holds when you slice by query intent and source before declaring one system better.

How do you apply NDCG when chunks are the retrieval unit?

Define relevance at the chunk level in your qrels and grade each retrieved chunk against the query, then compute NDCG@k over the ranked chunk list exactly as you would over documents. The math does not change; only the unit of relevance does.

Try Airbyte Agents

Airbyte connects your agents to all of your data and assembles context before they run. Build agents that actually know your business.

Try it free Talk to sales

NDCG Explained: How to Measure Ranking Quality in Retrieval Systems

Related posts

Try Airbyte Agents