Senior Evaluation AI Engineer

Key Responsibilities:

• Design and implement automated evaluation pipelines for multi-agent LLM systems.
• Develop custom metrics and grading approaches for domain-specific tasks.
• Build evaluation datasets and monitor system drift and observability.

Required Qualifications:

• 5-7 years in software engineering, test automation, or AI engineering.
• Strong Python skills and experience with LangChain, LangGraph, and LangSmith.
• Experience with adversarial testing and AI safety evaluation.

Preferred Qualifications:

• Experience with evaluation tools like Ragas, DeepEval, or Arize Phoenix.
• Familiarity with Bedrock Agents and integrating evaluations into CI/CD.