judgment-list-author

Related skills (elasticsearch-relevance-tests, opensearch-relevance-tests, vector-search-recall-tests) all require a judgment list - a set of (query, document_id, grade) triples that define what "relevant" means for your product. No automated metric creates that corpus. This skill does.

Per TREC's pooling methodology (opens in new window), "NIST pools the individual results, judges the retrieved documents for correctness, and evaluates the results" - the judgment list is the non-automated step that all automated metrics depend on.

When to use

• Before running _rank_eval or recall@k for the first time: no judgment list means no metrics.
• After a major schema or corpus change: existing judgments cover documents that may no longer exist or new documents that are entirely unjudged.
• When a relevance-regression check reports > 30% unrated docs across queries: the judgment pool is stale.
• When adding a new query segment (new product category, new language) with no coverage in the existing set.

Step 1 - Select the query set (head / torso / tail sampling)

Not all queries deserve equal judgment effort. Sample across three tiers:

Pull the sample from query logs, not from intuition. Use a 90-day window to avoid seasonal skew. Keep the raw log row for each sampled query - you will need it to assign traffic weight when computing weighted NDCG.

A starting corpus of roughly 200-400 queries across tiers gives sufficient coverage for NDCG-based gates. Per Quepid docs (opens in new window), "100 judgments (10 pages of 10 search results) is a good starting point and likely sufficient for most cases".

Step 2 - Choose the grading scale

Two options are widely used:

Binary (TREC standard)

Per TREC qrels format (opens in new window), the classic qrels file uses two values: 0 (not relevant) and 1 (relevant). Simple to collect; sufficient for Precision@k and Recall@k. Use binary when raters have low domain expertise or when the query intent is unambiguous (navigational queries).

Qrels file format (4 columns, per TREC qrels format (opens in new window)):

<topic_id> <iteration> <doc_id> <relevance>
1 0 doc-abc-123 0
1 0 doc-xyz-456 1

The iteration column is "almost always zero and not used" per TREC. Unjudged documents are assumed irrelevant in evaluation.

Graded 0-3 (Quepid default)

Per Quepid judgment rating best practices (opens in new window), the 0-3 scale maps to these grades and customer-feeling cues (the source frames them for e-commerce; restate them in your own domain's terms):

Use graded when you need NDCG or DCG (metrics that reward highly relevant results at higher ranks). Required by elasticsearch-relevance-tests Step 1 which uses a 4-point (0-3) scale.

Step 3 - Write grading guidelines for raters

Grading guidelines prevent scale drift between raters and across time. A minimal guidelines document covers:

1. Task definition: state the user's information need, not just the query string. "query: running shoes" could be navigational (find Nike page) or informational (compare models).
2. Grade anchors with examples: for each grade level, list 2-3 concrete examples from your domain. Raters calibrate to examples faster than to prose definitions.
3. Edge cases by domain: e-commerce (out-of-stock items, variant SKUs), docs search (older versions, deprecated APIs), support search (resolved vs open tickets).
4. What to do with unrateable documents: Quepid supports an unrateable flag per Quepid API (opens in new window) (permitted judgement params: rating, unrateable, judge_later, query_doc_pair_id, user_id, explanation). Use it for documents where the rater cannot determine relevance (e.g. page behind a login wall).

Store the guidelines in version control alongside the judgment file. When guidelines change, treat it as a new judgment round - old and new grades are not comparable.

Step 4 - Set up Quepid for rater workflow

Quepid (opens in new window) is the standard open-source tool for collaborative judgment authoring. It runs as a self-hosted Rails app and connects to Elasticsearch, OpenSearch, Solr, Algolia, and other backends.

Setup flow:

1. Deploy Quepid via Docker: docker-compose up per the Quepid README.
2. Create a Case - one Case = one judgment corpus for one search endpoint + query set.
3. Import queries (CSV or manual entry). Each query gets a displayed information-need description shown to the rater.
4. Assign raters. Quepid tracks user_id per judgment per Quepid API (opens in new window), enabling per-rater analysis.
5. Raters work through the Human Rating Interface: query + information need + rendered document + grade selector.
6. Export when done. Per Quepid API (opens in new window), the export is a CSV whose header row starts query, docid and then appends one column per judge name, with filename book_{id}_judgements.csv. JSON and Learning-to-Rank formats are also supported per Quepid docs (opens in new window).

For teams without Quepid, a spreadsheet works for small sets (under 500 judgments): columns query_id, query_text, doc_id, grade, rater_id, notes. Convert to qrels format before feeding _rank_eval.

Step 5 - Measure inter-rater agreement (Cohen's kappa)

Have at least two independent raters judge the same 10-20% overlap set. Compute Cohen's kappa to verify the scale is being applied consistently.

Formula per Cohen's kappa, Wikipedia (opens in new window):

kappa = (p_o - p_e) / (1 - p_e)

where p_o is observed agreement and p_e is expected chance agreement.

Interpretation thresholds (Landis and Koch, 1977, as cited in Cohen's kappa, Wikipedia (opens in new window)):

For binary judgments, kappa < 0.60 means your guidelines are ambiguous. Rewrite the edge-case section, run a calibration session with raters, and re-judge the overlap set before proceeding.

from sklearn.metrics import cohen_kappa_score

rater_a = [3, 2, 1, 0, 3, 2, 2, 1, 0, 3]
rater_b = [3, 2, 0, 0, 3, 1, 2, 1, 0, 2]

kappa = cohen_kappa_score(rater_a, rater_b)
print(f"Cohen's kappa: {kappa:.3f}")
# 0.61-0.80 = substantial agreement; proceed to full rating round

When multiple raters cover non-overlapping document sets (common for large judgment pools), use Krippendorff's alpha instead - it handles missing data across raters.

Step 6 - Pool from multiple retrieval systems

When you have results from more than one system (e.g. current BM25 + candidate neural re-ranker), use TREC-style depth-k pooling to maximize judgment coverage.

Per TREC pooling, Wikipedia (opens in new window), in pooling "the top-ranked n documents from each contributing run are aggregated, and the resulting document set is judged completely."

Practical pooling:

def pool_results(system_results: dict[str, list[str]], depth: int = 10) -> set[str]:
    """
    system_results: { system_name: [doc_id, ...] } top-depth per query
    Returns the union of all doc IDs in the pool.
    """
    pool = set()
    for docs in system_results.values():
        pool.update(docs[:depth])
    return pool

Pool depth = 10 is standard for small collections (< 100k docs). Increase to 20-50 when you have > 3 candidate systems to avoid missing relevant documents that only appear in one system's lower ranks.

Documents outside the pool are unjudged. Per TREC qrels format (opens in new window), documents "not occurring in the qrels file were not judged by the human assessor and are assumed to be irrelevant in the evaluations used in TREC." This is conservative but consistent across systems.

Step 7 - Establish refresh cadence

Judgment lists go stale when the document corpus changes substantially. Define explicit refresh triggers:

As a minimum, run a lightweight staleness check monthly: for each query, count the unrated_docs fraction in _rank_eval results. If the average exceeds 20%, schedule a re-judging session.

Output format

The judgment list consumed by elasticsearch-relevance-tests and opensearch-relevance-tests is a JSON array:

[
  {
    "query_id": "q001",
    "query": "running shoes",
    "ratings": [
      { "doc_id": "doc-abc", "rating": 3 },
      { "doc_id": "doc-xyz", "rating": 1 },
      { "doc_id": "doc-mno", "rating": 0 }
    ]
  }
]

Or equivalently as a TREC qrels file (4 columns) for binary cases, which tools like trec_eval and the Elasticsearch _rank_eval API both accept after mapping doc_id to the index's _id field.

Anti-patterns

Limitations

• TREC pooling assumes multiple competing systems. Single-system shops have shallower pools and higher unjudged-doc rates.
• Graded 0-3 scale requires domain-expert raters. Binary scale is appropriate when domain expertise is scarce.
• Cohen's kappa understates agreement on rare categories (e.g. grade 3 "perfect" is rare for tail queries). Per Cohen's kappa, Wikipedia (opens in new window), kappa "tends to underestimate the agreement on the rare category" - supplement with per-grade agreement counts when the kappa is borderline.
• Judgment effort scales with query count and pool depth. A 400-query set at depth 20 = 8,000 documents to judge; plan rater time accordingly.

References

• [TREC qrels format] - 4-column qrels file format, binary relevance scale, pooling convention: https://trec.nist.gov/data/qrels_eng/
• [TREC pooling, Wikipedia] - depth-k pooling methodology: https://en.wikipedia.org/wiki/Text_Retrieval_Conference
• [Quepid] - open-source judgment authoring tool (Rails, supports ES / OpenSearch / Solr / Algolia): https://github.com/o19s/quepid
• [Quepid judgment rating best practices] - 0-3 grade scale (Poor / Fair / Good / Perfect) with per-grade rater cues: https://github.com/o19s/quepid/wiki/Judgement-Rating-Best-Practices
• [Quepid docs] - rater workflow, 100-judgment baseline, export formats (CSV / JSON / LTR): https://quepid-docs.dev.o19s.com/2/quepid
• [Quepid API] - judgment fields (rating, unrateable, judge_later, explanation), CSV export structure book_{id}_judgements.csv: https://github.com/o19s/quepid/blob/main/app/controllers/api/v1/judgements_controller.rb
• [Cohen's kappa, Wikipedia] - kappa formula, Landis and Koch (1977) thresholds: https://en.wikipedia.org/wiki/Cohen%27s_kappa
• elasticsearch-relevance-tests - consumes judgment lists for _rank_eval
• opensearch-relevance-tests - consumes judgment lists for OpenSearch rank eval
• vector-search-recall-tests - consumes judgment lists for recall@k evaluation