Machine Learning Engineer, Evals

Responsibilities:

• Run the full eval pipeline end to end and reproduce known results during onboarding.
• Build a judge calibration protocol: sample human-labeled decisions, measure agreement, identify drift zones.
• Extend existing benchmarks with new tasks targeting known capability gaps, including the prompt, environment, rubric, grader, and QA.

Qualifications:

• 3+ years in software engineering, ML engineering, or data science with concrete evaluation experience.
• Experience with at least one LLM evaluation framework with real opinions on its strengths and weaknesses.
• Solid Python, comfort with Git, CI/CD, Docker, and Linux command line.

Preferred:

• RLVR/RLHF pipeline experience, training data curation, distributed eval orchestration.
• Benchmark design from scratch, red teaming, adversarial eval, psychometrics.