Senior AI Quality/Evaluation Engineer
Benefits
3w PTO
1w paternity
Similar Jobs
See allSenior AI Engineer
League
Canada
Python
MLOps
GCP
Senior AI Engineer
Osano
US
FastAPI
LangGraph
AI Agents
Manager, AI Operations & Evaluation
Chime
US
SQL
Python
Data Visualization
Applied AI Evaluation Scientist
Jump
US
Python
SQL
LLMs
Senior Machine Learning Engineer - Integrations (AU remote)
Canva
Australia
LLMs
Python
TypeScript
What You’ll Do:
- Design and build the evaluation infrastructure that ensures the platform's AI systems produce accurate, well-sourced, high-quality responses
- Build automated test suites that validate answer quality across agent pipeline changes
- Develop regression detection systems that catch quality degradation before it reaches users
What You Bring:
- 6+ years of software engineering experience, with significant work in testing infrastructure, ML evaluation, or quality systems
- Experience building evaluation or testing frameworks for LLM-based or ML-based systems
- Understanding of how to measure response quality for generative AI: accuracy, groundedness, citation correctness, relevance
What We Offer:
- 15 vacation days per year
- Flexible work options
- Health, dental, vision, and paramedical coverage
IDC
IDC is building the next generation of AI-powered intelligence platforms that transform how technology decisions get made. As the premier global provider of trusted technology intelligence, IDC equips business and technology leaders with the evidence they need to make confident decisions.