LLM evaluation, ML model testing, AI-assisted test generation and review, and search relevance.
AI-assisted test generation: LLM-written tests from natural-language specs with confidence scoring, spec coverage gap mapping, and model-based testing. Builds state-machine models and turns them into test suites, and adds adversarial review agents (ai-test-curator, ai-test-shallow-coverage-critic) that catch hallucinated APIs, weak assertions, and happy-path-only coverage.
LLM and prompt evaluation with Promptfoo, DeepEval and Ragas, Giskard vulnerability scans, Langfuse tracing, and golden-dataset regression suites. Audits eval suites for methodology errors. Agents (prompt-eval-reviewer, llm-red-team-planner) review an eval suite and plan adversarial probe campaigns across the OWASP LLM Top 10.
ML model testing: Giskard and Deepchecks validation, Evidently drift monitoring, Fairlearn fairness metrics, and model performance regression gates. Assigns risk tiers with the fairness and explainability evidence each tier requires, and wires Jupyter notebook CI with papermill, nbval, testbook and nbstripout.
Search relevance testing for Elasticsearch, OpenSearch, Solr, vector and hybrid search: judgment lists and NDCG, MRR and recall@k regression checks. Builds judgment lists with rater guidelines, benchmarks vector recall against latency, and measures fusion lift in hybrid pipelines. A review agent (relevance-regression-reviewer) compares before and after metrics per query.
