Source Job

$75–$175/hr

  • Lead benchmark design, validation, and grading operations for AI research projects.
  • Write rubrics, develop gold-standard answers, and analyze model failures.
  • Design rigorous model comparison studies and user studies on model interactions.

Python

20 jobs similar to Senior Evaluation Expert

Jobs ranked by similarity.

AI Evaluator

OpenAI
$30–$60/hr

  • Write questions and tasks that isolate and test specific model capabilities.
  • Develop rubrics that define strong, partial, and poor answers.
  • Grade model outputs against rubrics and explain scoring decisions.

Canada United States Unlimited PTO

  • Define and maintain the evaluation rubric for the AI agent across adherence, outcome, and conversation experience axes, and build scalable evaluation systems including LLM-based judges and regression suites.
  • Review and annotate production conversations weekly to identify failure modes, classify them by severity, and route findings to engineering and product teams.
  • Report quality trends to stakeholders with clear uncertainty, partner cross-functionally, and document methodology to ensure reproducibility.

Cresta unlocks the true potential of the customer experience with a unified AI platform combining conversational AI agents, real-time human augmentation, and conversation intelligence. Backed by over $270 million from top investors like a16z, Greylock, and Sequoia, Cresta's leadership includes former Google AI leaders, and the company serves major clients like United Airlines and Marriott.

US

  • Design state-of-the-art datasets that drive frontier model training and evaluation based on current model performance and academic partnerships.
  • Translate benchmark insights into clear, compelling narratives for customer-facing presentations, technical reports, and go-to-market materials.
  • Represent Snorkel's research externally through publications, blog posts, conference talks, and customer engagements.

Snorkel AI is the frontier AI data lab, helping teams build the data and environments behind high-performing frontier and agentic AI. Founded in 2019 out of Stanford AI Lab, it works with leading AI labs and enterprises to move from better data to better outcomes.

US

  • Red-team frontier benchmarks: identify and reproduce issues like reward hacking, grader failures, task ambiguity, shortcuts, and environment vulnerabilities.
  • Develop benchmark repairs: propose next steps, implement fixes with authors, and verify without breaking valid solutions.
  • Drive research impact: contribute to technical reports, open-source tools, publications, and new approaches to benchmark evaluation.

Snorkel AI is the frontier AI data lab, helping teams build the data and environments behind high-performing frontier and agentic AI. Founded out of the Stanford AI Lab in 2019, Snorkel works with leading AI labs and enterprises and is scaling rapidly.

$170,000–$220,000/yr
US

  • Own the context layer: manage what Mimo knows, when, and at what cost, with retrieval and memory as one budget.
  • Define tool contracts and build evaluation practice: schemas, results, failure modes, and regression gates for every change.
  • Shape how Mimo reasons, asks, explains, and recovers, and decide which model runs where.

MimoLabs builds Mimo, an agentic system for making physical things, from design to production across 3D printing, CNC, and laser cutting. We are a small, VC-funded, experienced team based in Detroit and remote across the US, fast, adaptable, and honest.

$175,000–$195,000/yr
US Canada Unlimited PTO 12w maternity 12w paternity

  • Build evaluation frameworks for AI agents, including datasets and metrics for real-world task performance.
  • Design experiments to compare models and tools, identifying regressions and trade-offs.
  • Investigate agent failures and improve performance across models, harnesses, and tooling.

Gauntlet builds the financial systems of the future, operating across the onchain finance stack to offer vault products. They serve over $1.7B in client TVL across fintechs, protocols, exchanges, and capital allocators, combining traditional finance and crypto-native expertise.

Global

  • Design, build, and maintain cybersecurity benchmarks for AI within evaluation infrastructure using Inspect.
  • Sandbox and harden evaluation environments to prevent AI escape and maintain security.
  • Integrate with AI providers to run benchmarks and collaborate with researchers for accurate, insightful outputs.

Epoch AI investigates trends in machine learning and the economic consequences of AI, developing a comprehensive knowledge base for policymakers. The research institute is committed to inclusive, equitable community and produces rigorous, accessible work on compute trends, data scarcity, and algorithmic progress.

$80–$150/hr

  • Design evaluations of conceptual reasoning in AI safety, covering governance, alignment, macrostrategy, and economics.
  • Develop rubrics that separate genuine reasoning from plausible-sounding filler.
  • Iterate on tasks with frontier models and collaborate with a top research team.

Trajectory Labs is a startup automating AI safety research. We build safety and alignment evals, red-teaming programs, and RL environments for frontier AI labs. As a one-year-old company, we've been featured in major frontier lab safety reports and led high-impact red-teaming efforts.

Global

  • Design and ship features end to end across data model, API, background workers, and UI.
  • Build and improve agent and model evaluation: sandboxed trials, scoring, and trustworthy aggregation of thousands of runs.
  • Improve benchmark task quality checks, integrate new models and agents, and keep the system fast and reliable as usage grows.

Turing accelerates superintelligence by providing high-quality datasets, reinforcement learning environments, and research benchmarks to frontier AI labs, and also deploys agentic AI systems for Fortune 500 enterprises. Headquartered in San Francisco, it operates with startup speed and an AI-forward culture.

$235,000–$376,000/yr
US Unlimited PTO

  • Identify and vet consequential opportunities across Data products, tools, AI measurement, and evaluation.
  • Develop and validate methods for evaluating generative AI and agentic workflows, building prototypes and tools.
  • Provide technical direction, mentor teams, and communicate findings to C-level executives.

Figma is on a mission to make design accessible to all, building collaborative tools for digital product development. The company is growing its team of passionate creatives and builders, fostering a culture that values diversity and equal opportunity.

Mexico Brazil Argentina

  • Architect and scale automated evaluation pipelines for multi-agent LLM systems.
  • Define domain-specific metrics using advanced evaluation techniques like LLM-as-a-judge.
  • Build benchmarking infrastructure and monitor live agent workflows for drift and cost.

This company develops an AI-driven platform that automates complex enterprise workflows for corporate real estate, including lease abstraction and portfolio optimization. The size and culture of the organization are not specified in the posting.

Global

  • Review and evaluate multiple business presentation slide decks
  • Compare presentations to determine which exhibits higher professional quality
  • Provide detailed reasoning for your comparative assessments verbally

Terac is building the world's largest pool of vetted human experts for AI. Researchers, AI labs, and product teams use Terac to recruit, screen, and pay study participants across industries, languages, and skill sets.

Global 7w PTO

  • Build and own evaluation infrastructure for Circle's AI agents.
  • Diagnose quality breakdowns across the agent pipeline.
  • Grow datasets and lead the AI Quality engineering team.

Circle builds an AI-powered all-in-one platform for digital businesses, uniting audiences with discussions, live streams, events, chat, courses, and payments. It's a fully remote team of about 270 people from 30+ countries, with an async culture and annual global offsites.

$160,000–$195,000/yr
US

  • Pick the right model for each job, from frontier APIs to open-weight models, and know when fine-tuning beats a better prompt.
  • Build evaluation harnesses and benchmarks that tell us whether a change made things better, then take models to serving and tune cost, latency, and quality.
  • Sit with the client's team, find where AI moves their numbers, and say no to work that won't pay back.

Lunon is an AI-native consulting firm in New York City, backed by General Catalyst, that embeds engineers inside businesses to ship AI software that runs without them. The team is young, fast-moving, and owns what it ships end to end.

US

  • Own quality for non-deterministic, multi-agent AI systems, building the evaluation harness that underpins AI governance.
  • Verify grounding and faithfulness, building automated checks to stop hallucinations from reaching production.
  • Red-team for prompt injection, jailbreaks, and tool misuse, and prove multi-tenant data isolation.

SOFICO Intelligence uses AI to extract value from back-office systems, building agentic AI for financial, leasing, fleet, and mobility data. Backed by global ERP leader SOFICO, it's a remote-first team of curious, self-directed builders.

APAC

  • Design end-to-end LLM evaluation plans for business scenarios such as dialogue and financial trading.
  • Build evaluation metric systems, rubrics, and high-quality datasets with credible, reproducible conclusions.
  • Analyze model limitations, automate evaluation workflows, and collaborate to drive model improvement.

Binance is a leading global blockchain ecosystem behind the world's largest cryptocurrency exchange by trading volume and registered users. Trusted by over 300 million people in more than 100 countries, the company offers a fast-paced, results-driven culture with a flat structure and world-class talent.

$200,000–$300,000/yr
United States

  • Lead the development of a rigorous evaluation framework for enterprise AI agents, including experimental methodology and production-quality tools.
  • Design and execute statistically sound experiments to assess planner models, execution models, and inference optimization strategies.
  • Provide technical leadership, establish evaluation standards, and deliver actionable recommendations supporting enterprise deployment decisions.

Jobgether is an AI-powered job matching platform that connects candidates with hiring companies. It uses technology to match applications and share top-fitting candidates with employers.

$100,000–$150,000/yr
US

  • Design architectures for specialized subagents in live customer conversations, covering technical Q&A, product pricing, discovery, objection handling, and more.
  • Master the dialogue platform, train Superhumans using prompting, RAG, and fine-tuning, and prototype complex prompting workflows.
  • Build routing systems that decide when to answer directly, invoke subagents, escalate to humans, or decline unsafe actions.

1mind is building enterprise software for Autonomous Customer Experience, deploying AI 'Superhumans' that serve buyers across the entire journey. The company is remote-first with a fast-moving culture emphasizing ownership, autonomy, and impact from day one.

$180,000–$260,000/yr
US

  • Build repeatable benchmarks and evaluation pipelines for model accuracy, hallucinations, retrieval quality, latency, and cost.
  • Turn research prototypes into production retrieval, drafting, document, and agent workflows.
  • Develop backend services, APIs, and internal tools to support testing and releases.

They are a healthtech company building dependable AI products for legal work. Their team is product-focused and works remotely across the United States.