Similar Jobs
See allAI Evaluation Engineer
Undisclosed
Global
Python
TypeScript
LLM Evaluation
AI Engineer
Elicit
US
Python
Evaluation Design
Software Engineering
AI Engineer
Mastra
North America
TypeScript
AI Agents
LLM Evaluation
Software Engineering Director, Agentic Evaluations
Partner Company
US
Python
TypeScript
Agentic AI
Senior Software Engineer – LLM Evaluation
Partner Company
US
Python
JavaScript
ReactJS
Role Overview:
- Design and build evaluation infrastructure for production-grade AI agents.
- Develop automated testing systems for complex reasoning and multi-turn conversations.
- Own the evaluation architecture from ground up in a fully remote environment.
Key Responsibilities:
- Build CI/CD testing pipelines to validate AI agent behavior.
- Implement adversarial testing scenarios for security vulnerabilities.
- Create statistical evaluation frameworks for non-deterministic AI behavior.
Requirements:
- Strong proficiency in Python or TypeScript.
- Experience building LLM evaluation pipelines and integrating tests into CI/CD.
- Excellent English and ability to overlap with US Central Time business hours.
Benefits:
- Fully remote work with flexible schedule.
- Paid training and professional development opportunities.
- Performance-based annual pay increases and negotiable hourly rate.
Jobgether
A company building production-grade AI agents and systems. They operate in a fully remote, globally distributed environment with a culture focused on innovation, autonomy, and continuous improvement.