qa-search-relevance
Search relevance testing: 6 skills (elasticsearch-relevance-tests, hybrid-search-eval-author, judgment-list-author, opensearch-relevance-tests, solr-relevance-tests, vector-search-recall-tests) and 1 agent (relevance-regression-reviewer). IR-metrics-driven NDCG / MRR / Recall@k regression detection.
Install this plugin
/plugin install qa-search-relevance@testland-qaPart of role bundle: qa-role-ai
qa-search-relevance
IR-metrics-driven search relevance testing - judgment lists, NDCG / MRR / Recall@k, vector recall@k vs latency Pareto curve. Four skills + one reviewer agent that synthesizes per-query regression analysis across term-based and vector search.
Components
| Type | Name | Description |
|---|---|---|
| Skill | elasticsearch-relevance-tests | Engine umbrella for Elasticsearch, OpenSearch, and Apache Solr: _rank_eval API; judgment lists; metrics (Precision@K, Recall@K, MRR, DCG/NDCG, ERR); per-query regression detection; Quepid + Splainer integration; per-engine references for the OpenSearch delta (Workbench, neural, hybrid, migration parity) and the Solr delta (LTR, debugQuery, eDisMax, external nDCG) |
| Skill | vector-search-recall-tests | Brute-force ground truth; recall@k vs latency Pareto; HNSW M / ef_construct / ef sweep; embedding-model-upgrade drift; ANN-Benchmarks framework |
| Skill | judgment-list-author | Bootstrap human-judgment ground-truth lists: query sampling, grading scales, kappa, Quepid, pooling. |
| Skill | hybrid-search-eval-author | Evaluate hybrid retrieval (BM25 + vector + reranker) with RRF fusion and nDCG/MRR. |
| Agent | relevance-regression-reviewer | Adversarial reviewer; per-query regression detection; refuses when head queries drop > 0.05 OR when judgments are stale (> 50% unrated) |
Install
/plugin marketplace add testland/qa
/plugin install qa-search-relevance@testland-qaSkills
elasticsearch-relevance-tests
Author search-engine relevance regression tests for Elasticsearch, OpenSearch, and Apache Solr. Core workflow on the Elasticsearch Ranking Evaluation API (`POST {index}/_rank_eval`) - judgment lists (query + expected docs at ranks), per-query metrics (Precision@K, Recall@K, MRR, DCG, ERR), reproducible test corpora; pair with Quepid + Splainer for interactive judgment authoring. Per-engine references cover the OpenSearch delta (Search Relevance Workbench, neural query DSL, hybrid BM25 + neural pipelines, ES-to-OS migration parity) and the Apache Solr delta (no _rank_eval: debugQuery score explain, LTR feature/model store REST, eDisMax qf/pf/mm tuning, external nDCG harness). Use before changing analyzers, synonyms, boosts, or query templates on an Elasticsearch, OpenSearch, or Solr index that serves user-facing search, so the NDCG / MRR baseline is captured first.
hybrid-search-eval-author
Evaluates hybrid retrieval pipelines (BM25 + vector + reranker) end-to-end: authors ground-truth judgment sets, computes nDCG@k and MRR over fused results, measures the lift from Reciprocal Rank Fusion vs weighted fusion vs single-stage retrieval, and quantifies reranker (cross-encoder/Cohere/bge) impact. Use when a production system combines lexical and semantic retrieval and you need a numeric relevance baseline, fusion-strategy comparison, or evidence that a reranker is earning its latency cost.
judgment-list-author
Bootstraps human-relevance judgment lists (query sets, grading scales, rater guidelines, inter-rater agreement, Quepid tooling, TREC-style pooling, and refresh cadence) that serve as ground truth for search-relevance test suites. Use when a team needs to create or refresh the judgment corpus before running NDCG / MRR / Recall@k evaluations.
vector-search-recall-tests
Vector search benchmarking - recall@k vs latency tradeoffs, ground-truth construction via brute-force, HNSW tuning (M / ef_construct / ef per Qdrant docs), embedding-model-upgrade drift detection. Use ANN-Benchmarks framework for cross-engine comparison; per-engine clients (Qdrant, Weaviate, pgvector, Pinecone, Elasticsearch k-NN, Milvus) for in-product tests. Use when HNSW / IVF parameters are being tuned or an embedding model is swapped, and recall@k on the existing corpus has never been measured against brute-force ground truth.