Similar Jobs
See allAI Evaluator
OpenAI
Python
Data Analysis
AI Agent Evaluation Engineer
Cresta
Canada
Python
LLM APIs
Data Analysis
Research Lead, Open Benchmarks Grants
Snorkel AI
US
NLP
Experimental Design
Communication
Research Scientist, Open Benchmarks Red Team
Snorkel AI
US
Machine Learning
Python
AI Evaluation
Staff AI Engineer, Agents and Evals
MimoLabs
US
Python
Role Overview:
- Lead benchmark design and validation for AI research projects.
- Guide grading operations including calibrating graders and resolving disputes.
Key Responsibilities:
- Design scope, question types, difficulty levels, and scoring methods for benchmarks.
- Write and review rubrics to ensure consistent grading.
- Develop gold-standard answers and test sets.
Qualifications:
- At least 5 years of experience in evaluation or domain expertise.
- Experience designing assessments or benchmarks used by others.
- Proficiency in Python and evaluation tools.
Surge AI
Surge AI connects senior evaluation experts to support AI research and model testing. The company focuses on rigorous benchmark design and grading for AI systems, with a network of specialized contractors working on diverse projects.