Source Job

Canada United States Unlimited PTO

  • Define and maintain the evaluation rubric for the AI agent across adherence, outcome, and conversation experience axes, and build scalable evaluation systems including LLM-based judges and regression suites.
  • Review and annotate production conversations weekly to identify failure modes, classify them by severity, and route findings to engineering and product teams.
  • Report quality trends to stakeholders with clear uncertainty, partner cross-functionally, and document methodology to ensure reproducibility.

Python LLM APIs Data Analysis

18 jobs similar to AI Agent Evaluation Engineer

Jobs ranked by similarity.

$75–$175/hr

  • Lead benchmark design, validation, and grading operations for AI research projects.
  • Write rubrics, develop gold-standard answers, and analyze model failures.
  • Design rigorous model comparison studies and user studies on model interactions.

Surge AI connects senior evaluation experts to support AI research and model testing. The company focuses on rigorous benchmark design and grading for AI systems, with a network of specialized contractors working on diverse projects.

$170,000–$220,000/yr
US

  • Own the context layer: manage what Mimo knows, when, and at what cost, with retrieval and memory as one budget.
  • Define tool contracts and build evaluation practice: schemas, results, failure modes, and regression gates for every change.
  • Shape how Mimo reasons, asks, explains, and recovers, and decide which model runs where.

MimoLabs builds Mimo, an agentic system for making physical things, from design to production across 3D printing, CNC, and laser cutting. We are a small, VC-funded, experienced team based in Detroit and remote across the US, fast, adaptable, and honest.

AI Evaluator

OpenAI
$30–$60/hr

  • Write questions and tasks that isolate and test specific model capabilities.
  • Develop rubrics that define strong, partial, and poor answers.
  • Grade model outputs against rubrics and explain scoring decisions.

$100,000–$150,000/yr
US

  • Design architectures for specialized subagents in live customer conversations, covering technical Q&A, product pricing, discovery, objection handling, and more.
  • Master the dialogue platform, train Superhumans using prompting, RAG, and fine-tuning, and prototype complex prompting workflows.
  • Build routing systems that decide when to answer directly, invoke subagents, escalate to humans, or decline unsafe actions.

1mind is building enterprise software for Autonomous Customer Experience, deploying AI 'Superhumans' that serve buyers across the entire journey. The company is remote-first with a fast-moving culture emphasizing ownership, autonomy, and impact from day one.

US

  • Own the end-to-end product lifecycle of Joy.ai's AI agent reasoning layer, from discovery to launch.
  • Drive agent orchestration across skills, tools, and other agents within Asurion's ecosystem.
  • Define product-quality evals that prove the agent is useful, trustworthy, and ready for real customers.

Asurion is a global tech care leader providing protection, repair, installation, and expert support for consumer devices. Its products and services help nearly 300 million customers worldwide, with a culture that values customer-first, teamwork, and integrity.

Mexico Brazil Argentina

  • Architect and scale automated evaluation pipelines for multi-agent LLM systems.
  • Define domain-specific metrics using advanced evaluation techniques like LLM-as-a-judge.
  • Build benchmarking infrastructure and monitor live agent workflows for drift and cost.

This company develops an AI-driven platform that automates complex enterprise workflows for corporate real estate, including lease abstraction and portfolio optimization. The size and culture of the organization are not specified in the posting.

Global 7w PTO

  • Build and own evaluation infrastructure for Circle's AI agents.
  • Diagnose quality breakdowns across the agent pipeline.
  • Grow datasets and lead the AI Quality engineering team.

Circle builds an AI-powered all-in-one platform for digital businesses, uniting audiences with discussions, live streams, events, chat, courses, and payments. It's a fully remote team of about 270 people from 30+ countries, with an async culture and annual global offsites.

US

  • Own quality for non-deterministic, multi-agent AI systems, building the evaluation harness that underpins AI governance.
  • Verify grounding and faithfulness, building automated checks to stop hallucinations from reaching production.
  • Red-team for prompt injection, jailbreaks, and tool misuse, and prove multi-tenant data isolation.

SOFICO Intelligence uses AI to extract value from back-office systems, building agentic AI for financial, leasing, fleet, and mobility data. Backed by global ERP leader SOFICO, it's a remote-first team of curious, self-directed builders.

$247,000–$363,000/yr
US Unlimited PTO

  • Architect across the AI ecosystem to ensure investments compose instead of collide, driving build-vs-buy decisions for PDLC-critical agents.
  • Define what an agent is at Life360 as a reusable unit, setting standards for safety, eval, and authZ, and build the shared infrastructure.
  • Build hands-on by shipping production code against the highest-leverage components, mentoring engineers across teams, and writing architecture decision records.

Life360's mission is to keep people close to the ones they love through a mobile app, Tile tracking devices, and Pet GPS tracker. With approximately 107.7 million monthly active users across over 180 countries, the company has more than 500 remote-first employees and fosters a culture of high integrity and direct communication.

Global

  • Design and ship features end to end across data model, API, background workers, and UI.
  • Build and improve agent and model evaluation: sandboxed trials, scoring, and trustworthy aggregation of thousands of runs.
  • Improve benchmark task quality checks, integrate new models and agents, and keep the system fast and reliable as usage grows.

Turing accelerates superintelligence by providing high-quality datasets, reinforcement learning environments, and research benchmarks to frontier AI labs, and also deploys agentic AI systems for Fortune 500 enterprises. Headquartered in San Francisco, it operates with startup speed and an AI-forward culture.

Brazil

  • Govern end-to-end software delivery value stream for an AI Content Support Agent.
  • Coordinate complex human-agent collaborative workflows and set clear autonomy boundaries.
  • Elevate AI fluency of cross-functional team and manage scope to ensure core deliverables.

CI&T helps large enterprises transform the potential of AI into real business impact with AI Deployment, AI-native execution, and tech-integrated business solutions. With 30 years of experience and 8,000 CI&Ters across more than 25 countries, they collaborate to build solutions with real impact, with AI integrated into work every day.

$175,000–$195,000/yr
US Canada Unlimited PTO 12w maternity 12w paternity

  • Build evaluation frameworks for AI agents, including datasets and metrics for real-world task performance.
  • Design experiments to compare models and tools, identifying regressions and trade-offs.
  • Investigate agent failures and improve performance across models, harnesses, and tooling.

Gauntlet builds the financial systems of the future, operating across the onchain finance stack to offer vault products. They serve over $1.7B in client TVL across fintechs, protocols, exchanges, and capital allocators, combining traditional finance and crypto-native expertise.

$160,000–$190,000/yr
Canada UK Ireland 2w PTO

  • Assess new agent capabilities, protocols, models, and infrastructure to decide which ones provide real value for merchants.
  • Build working prototypes and reference code to demonstrate what works with the product, handling handover to product or engineering partners.
  • Represent Clearer.io at events like Shopify DotDev, build relationships with platform teams, and turn insights into product proposals.

Clearer.io helps merchants achieve clarity in eCommerce by connecting to their existing stack and using AI agents to enrich catalogues, learn from reviews, and put intelligence to work inside tools they already own. The company is a global team backed by PSG Equity and Silversmith Capital Partners, with offices in Australia, the UK, Germany, Vietnam, and the US, focusing on listening first, building second, and shipping quickly.

$275,000–$300,000/yr
US Europe

  • Set technical direction for production agent systems, including orchestration, tool use, memory management, and failure recovery.
  • Build and operate shared foundations: evaluation frameworks, observability, execution controls, and model access with routing and failover.
  • Work with security and platform teams to ensure scoped permissions, data isolation, audit trails, and safe execution of tools and code.

Cobalt runs AI agents in production that test real customer applications, operating at the forefront of the pentesting industry. The company is rapidly expanding, offering a collaborative culture with mentorship, competitive compensation, and a focus on growth.

PM

Confidential
$160,000–$220,000/yr
US

  • Own customer outcomes and the metrics that prove them
  • Run deployments end to end with forward deployed engineers
  • Design, test and refine AI agents with the team

They are a late-stage, venture-backed enterprise AI company that helps customers deploy AI agents at scale. The company is growing quickly and values hands-on ownership, technical thinking, and a bias to action.

APAC

  • Design end-to-end LLM evaluation plans for business scenarios such as dialogue and financial trading.
  • Build evaluation metric systems, rubrics, and high-quality datasets with credible, reproducible conclusions.
  • Analyze model limitations, automate evaluation workflows, and collaborate to drive model improvement.

Binance is a leading global blockchain ecosystem behind the world's largest cryptocurrency exchange by trading volume and registered users. Trusted by over 300 million people in more than 100 countries, the company offers a fast-paced, results-driven culture with a flat structure and world-class talent.

Global

  • Design, build, and maintain cybersecurity benchmarks for AI within evaluation infrastructure using Inspect.
  • Sandbox and harden evaluation environments to prevent AI escape and maintain security.
  • Integrate with AI providers to run benchmarks and collaborate with researchers for accurate, insightful outputs.

Epoch AI investigates trends in machine learning and the economic consequences of AI, developing a comprehensive knowledge base for policymakers. The research institute is committed to inclusive, equitable community and produces rigorous, accessible work on compute trends, data scarcity, and algorithmic progress.

$200,000–$300,000/yr
United States

  • Lead the development of a rigorous evaluation framework for enterprise AI agents, including experimental methodology and production-quality tools.
  • Design and execute statistically sound experiments to assess planner models, execution models, and inference optimization strategies.
  • Provide technical leadership, establish evaluation standards, and deliver actionable recommendations supporting enterprise deployment decisions.

Jobgether is an AI-powered job matching platform that connects candidates with hiring companies. It uses technology to match applications and share top-fitting candidates with employers.