Testland
Skills & AgentsTopicsBlogAbout
Browse all skills & agents

opensearch-relevance-tests

Author OpenSearch relevance tests with Search Relevance Workbench (judgment lists, query sets, experiments), `_rank_eval` API (Elasticsearch-fork-compatible), and hybrid BM25 + neural ranking eval. Reuse Elasticsearch judgment list format; document the differences (neural search query DSL, hybrid weighting via `neural_query_enricher`). Use when an OpenSearch index turns on neural or hybrid search, or when a move off Elasticsearch has to prove relevance parity between the two clusters.

Install with skills.sh (any agent)

npx skills add testland/qa --skill opensearch-relevance-tests
View source

opensearch-relevance-tests

Per the OpenSearch search-relevance docs (opens in new window), _rank_eval is Elasticsearch-fork-compatible. The OpenSearch-specific surfaces worth testing: neural search, hybrid query, and the Search Relevance Workbench UI.

When to use

  • Team standardized on OpenSearch (often AWS shops, often migrated from Elasticsearch ≤ 7.10).
  • Adopting OpenSearch's neural search or hybrid search features.
  • Migration test between Elasticsearch and OpenSearch - relevance parity must hold.

Step 1 - Reuse judgment list format

OpenSearch's _rank_eval accepts the same JSON as Elasticsearch's. See elasticsearch-relevance-tests Step 1 for judgment list format + sourcing patterns. The CSV (query, doc_id, rating) schema is reusable.

A judgment is (query, doc_id, rating) on a 4-point scale: 0 = irrelevant, 1 = somewhat, 2 = relevant, 3 = highly relevant.

query,doc_id,rating
"running shoes",sku-1234,3
"running shoes",sku-5678,2
"running shoes",sku-9999,0
"red dress",sku-2222,3

Sourcing patterns:

SourceMethod
Query logs + click dataClick model (clicked = ≥1, multi-click = ≥2)
Search Relevance WorkbenchPairwise judgment UI + bulk import (Step 3)
Quepid (open source)Interactive UI for judges to rate per-query results
SplainerDiagnose why a doc ranked where it did
Domain SMEsHigh-stakes queries; manual rating

Each CSV row becomes one entry in the per-query ratings array of the _rank_eval request body in Step 2.

Step 2 - Submit _rank_eval request

POST products/_rank_eval
{
  "requests": [
    {
      "id": "running_shoes",
      "request": { "query": { "match": { "name": "running shoes" } } },
      "ratings": [
        { "_index": "products", "_id": "sku-1234", "rating": 3 }
      ]
    }
  ],
  "metric": { "dcg": { "k": 10, "normalize": true } }
}

Endpoint + metrics identical to Elasticsearch (per the OpenSearch search-relevance docs (opens in new window)).

Step 3 - Search Relevance Workbench

Per the OpenSearch search-relevance docs (opens in new window), the Search Relevance Workbench plugin (UI in OpenSearch Dashboards) provides:

  • Query Set Management - group queries logically (e.g., "head queries", "long-tail queries").
  • Judgment management - pairwise UI for judges + bulk import.
  • Experiments - run query-template A/B against the same judgment list; compare metric scores side-by-side.

Workbench experiments are the easiest pre-tuning baseline-and-compare workflow.

Step 4 - Neural search query

OpenSearch supports k-NN vector search natively. Test setup:

PUT my_index
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 768,
        "method": { "name": "hnsw", "engine": "lucene" }
      },
      "title": { "type": "text" }
    }
  }
}

Query:

POST my_index/_search
{
  "query": {
    "neural": {
      "embedding": {
        "query_text": "running shoes for marathon",
        "model_id": "<sentence-transformer-model>",
        "k": 10
      }
    }
  }
}

Test that neural results meet a recall@10 target against a held-out ground truth set:

def test_neural_recall_at_10():
    ground_truth = load_ground_truth("tests/marathon_queries.json")
    for query in ground_truth["queries"]:
        results = neural_search(query["text"], k=10)
        retrieved_ids = {r["_id"] for r in results}
        relevant_ids = set(query["relevant_ids"])
        recall = len(retrieved_ids & relevant_ids) / len(relevant_ids)
        assert recall >= 0.85, f"Recall {recall:.2f} below 0.85 for query: {query['text']}"

Pair with vector-search-recall-tests for HNSW parameter tuning.

Step 5 - Hybrid (BM25 + neural)

POST my_index/_search?search_pipeline=hybrid_pipeline
{
  "query": {
    "hybrid": {
      "queries": [
        { "match": { "title": "running shoes" } },
        { "neural": { "embedding": { "query_text": "running shoes", "k": 10 } } }
      ]
    }
  }
}

Hybrid weighting set up via search pipeline:

PUT _search/pipeline/hybrid_pipeline
{
  "phase_results_processors": [
    {
      "normalization-processor": {
        "normalization": { "technique": "min_max" },
        "combination": {
          "technique": "arithmetic_mean",
          "parameters": { "weights": [0.3, 0.7] }
        }
      }
    }
  ]
}

Test that hybrid weights matter:

def test_hybrid_weight_change_shifts_results():
    bm25_heavy_results = search_with_pipeline("hybrid_pipeline_03_07")  # 0.3 BM25 / 0.7 neural
    neural_heavy_results = search_with_pipeline("hybrid_pipeline_07_03")
    assert bm25_heavy_results != neural_heavy_results

Step 6 - Per-query metric regression (same as ES)

def test_no_query_drops_more_than_10_percent():
    current = rank_eval(judgments)
    baseline = json.loads(Path("tests/baseline-os.json").read_text())

    for q_id, baseline_entry in baseline["details"].items():
        current_score = current["details"][q_id]["metric_score"]
        delta = current_score - baseline_entry["metric_score"]
        assert delta >= -0.10, f"{q_id} dropped {delta:.2f}"

Step 7 - ES → OS migration parity test

Run the same judgment list against both clusters; metric scores should be within ε:

def test_es_os_parity():
    es_score = rank_eval_against("http://es:9200/products", judgments)
    os_score = rank_eval_against("http://os:9200/products", judgments)
    delta = abs(es_score - os_score)
    assert delta < 0.05, f"ES vs OS NDCG diff {delta:.2f} > 0.05"

If the index settings (analyzers, mappings) are identical, scores should match. Differences point to subtle config drift.

Anti-patterns

Anti-patternWhy it failsFix
Test only BM25 path when neural enabledNeural regression slips silentlyStep 4 + Step 5
Use neural without warm-up for testsCold cache → flaky latency testsWarm before measuring
Set hybrid weights without testing both extremesSubtle BM25/neural balance change shipsStep 5
Skip migration parity testOS deviation from ES surfaces in prodStep 7
Trust default analyzers across ES/OSSubtle stemmer differencesPin analyzer config

Limitations

  • Workbench UI is OpenSearch-Dashboards-only; for pure-CLI workflows, drive judgments + experiments via API.
  • OpenSearch's neural search requires model deployment via the ML Commons plugin; setup steps differ from raw _rank_eval.
  • API surface evolves; verify per the current OpenSearch search-relevance docs (opens in new window) for new fields.
  • Hybrid pipeline normalization techniques (min_max, l2) affect scores significantly; pin in CI.

References

Related skills

elasticsearch-relevance-tests

Author Elasticsearch relevance regression tests using the Ranking Evaluation API (`POST {index}/_rank_eval`) - judgment lists (query + expected docs at ranks), per-query metrics (Precision@K, Recall@K, MRR, DCG, ERR), reproducible test corpora; pair with Quepid + Splainer for interactive judgment authoring. Use before changing analyzers, synonyms, boosts, or query templates on an Elasticsearch index that serves user-facing search, so the NDCG / MRR baseline is captured first.

hybrid-search-eval-author

Evaluates hybrid retrieval pipelines (BM25 + vector + reranker) end-to-end: authors ground-truth judgment sets, computes nDCG@k and MRR over fused results, measures the lift from Reciprocal Rank Fusion vs weighted fusion vs single-stage retrieval, and quantifies reranker (cross-encoder/Cohere/bge) impact. Use when a production system combines lexical and semantic retrieval and you need a numeric relevance baseline, fusion-strategy comparison, or evidence that a reranker is earning its latency cost.

judgment-list-author

Bootstraps human-relevance judgment lists (query sets, grading scales, rater guidelines, inter-rater agreement, Quepid tooling, TREC-style pooling, and refresh cadence) that serve as ground truth for search-relevance test suites. Use when a team needs to create or refresh the judgment corpus before running NDCG / MRR / Recall@k evaluations.

solr-relevance-tests

Tests Apache Solr search relevance by querying a test core, asserting ranking and score expectations, uploading LTR feature stores and models via the `/schema/feature-store` and `/schema/model-store` REST APIs, using `debugQuery` for per-document score explain, tuning eDisMax parameters (`qf`, `pf`, `mm`, `bq`), and computing judgment-driven nDCG checks against pinned corpora. Use when the search stack runs Apache Solr (enterprise, SolrCloud, or embedded) and you need a pre-deploy relevance gate or LTR model verification.

vector-search-recall-tests

Vector search benchmarking - recall@k vs latency tradeoffs, ground-truth construction via brute-force, HNSW tuning (M / ef_construct / ef per Qdrant docs), embedding-model-upgrade drift detection. Use ANN-Benchmarks framework for cross-engine comparison; per-engine clients (Qdrant, Weaviate, pgvector, Pinecone, Elasticsearch k-NN, Milvus) for in-product tests. Use when HNSW / IVF parameters are being tuned or an embedding model is swapped, and recall@k on the existing corpus has never been measured against brute-force ground truth.