Similar Jobs
See allAI Evaluation Engineer
Jobgether
Global
Python
TypeScript
LLM Evaluation
AI Engineer
Mastra
North America
TypeScript
AI Agents
LLM Evaluation
Software Engineering Director, Agentic Evaluations
Partner Company
US
Python
TypeScript
Agentic AI
Senior CX Platform Engineer
Jobgether
US
Python
TypeScript
AWS
AI Engineer
Elicit
US
Python
Evaluation Design
Software Engineering
Role Overview:
- Design and build evaluation infrastructure for production-grade AI agents.
- Own the architecture from the ground up in a remote, global team.
Key Responsibilities:
- Build CI/CD pipelines, golden datasets, and adversarial testing for AI reliability.
- Implement statistical evaluation, observability, and structured output validation.
Requirements:
- Strong Python/TypeScript and LLM evaluation experience.
- CI/CD integration and independent architecture ownership skills.