Applied AI Evaluation Scientist

Agentic RAG Pipeline Evaluation & Optimization:

• Design evaluation datasets (synthetic query-answer pairs, adversarial cases, real user query sets).
• Measure retrieval quality using Recall@k, Precision@k, MRR, NDCG@k; assess appropriateness per use case.
• Evaluate and optimize chunking strategies; benchmark embedding models and re-rankers.

Broader AI/ML Evaluation:

• Conduct systematic error analysis through trace reading and failure mode identification.
• Design and validate LLM-as-Judge evaluators, refining iteratively and measuring TPR/TNR.
• Build and maintain golden datasets for CI regression testing of AI pipelines.

Collaboration & Data Review:

• Partner with Product to translate product requirements into measurable evaluation criteria.
• Partner with Engineering to instrument pipelines for observability and integrate evaluation checks into CI/CD.
• Lead or facilitate annotation workflows, measure inter-annotator agreement (Cohen's Kappa), and produce labeled datasets.