Similar Jobs

See all

Evaluation Design & Quality Definition:

  • Define and maintain the rubric for agent quality across adherence, outcome, and conversation experience.
  • Write annotation guidelines and failure taxonomy, run calibration sessions, and measure inter-rater agreement.
  • Classify and prioritize failure modes to guide engineering and product work.

Scalable Evaluation Systems:

  • Build and operate the evaluation pipeline scoring every production conversation with automated checks and LLM judges.
  • Calibrate automated judges against human labels and design sampling strategies for accurate quality reporting.
  • Maintain golden set and regression suite for validating prompt, model, and flow changes.

Coverage & Edge Cases:

  • Maintain a failure-mode coverage map and close gaps in testing.
  • Design evaluations for high-stakes behaviors like handoffs, misuse handling, and resilience under failures.
  • Define quality and latency thresholds the agent must meet.

Cresta

Cresta unlocks the true potential of the customer experience with a unified AI platform combining conversational AI agents, real-time human augmentation, and conversation intelligence. Backed by over $270 million from top investors like a16z, Greylock, and Sequoia, Cresta's leadership includes former Google AI leaders, and the company serves major clients like United Airlines and Marriott.

Apply for This Position