Source Job

AI Evaluator

OpenAI
$30–$60/hr

  • Write questions and tasks that isolate and test specific model capabilities.
  • Develop rubrics that define strong, partial, and poor answers.
  • Grade model outputs against rubrics and explain scoring decisions.

Python Data Analysis

20 jobs similar to AI Evaluator

Jobs ranked by similarity.

$75–$175/hr

  • Lead benchmark design, validation, and grading operations for AI research projects.
  • Write rubrics, develop gold-standard answers, and analyze model failures.
  • Design rigorous model comparison studies and user studies on model interactions.

Surge AI connects senior evaluation experts to support AI research and model testing. The company focuses on rigorous benchmark design and grading for AI systems, with a network of specialized contractors working on diverse projects.

$17–$17/hr
Global

  • Submit a real question you would actually ask an AI, typed naturally in your own words.
  • Tell why you needed to know and list two or three things a good answer must cover.
  • Earn $0.55 per accepted submission; no interview, no minimum hours, fully remote.

Kled is building a consumer app that pays people to upload their real photos, videos, and documents, then licenses that rights-aware data to AI labs and enterprises. Since 2026, it has scaled to 600,000+ active contributors, processes 5M uploads per day, and raised $15M+ from top investors.

Auditor

Anyone AI
$80–$150/hr
Global

  • Design audit tasks involving risk assessment, materiality, internal controls, substantive testing, and sampling.
  • Develop reference solutions that document procedures, evidence, findings, and justified conclusions.
  • Evaluate agents on supporting documents, transaction testing, workpapers, and professional skepticism.

Anyone AI develops AI agents to perform audit workflows and exercise evidence-based judgment. They operate as a remote, project-based company with flexible hours for contractors.

Canada United States Unlimited PTO

  • Define and maintain the evaluation rubric for the AI agent across adherence, outcome, and conversation experience axes, and build scalable evaluation systems including LLM-based judges and regression suites.
  • Review and annotate production conversations weekly to identify failure modes, classify them by severity, and route findings to engineering and product teams.
  • Report quality trends to stakeholders with clear uncertainty, partner cross-functionally, and document methodology to ensure reproducibility.

Cresta unlocks the true potential of the customer experience with a unified AI platform combining conversational AI agents, real-time human augmentation, and conversation intelligence. Backed by over $270 million from top investors like a16z, Greylock, and Sequoia, Cresta's leadership includes former Google AI leaders, and the company serves major clients like United Airlines and Marriott.

$170,000–$220,000/yr
US

  • Own the context layer: manage what Mimo knows, when, and at what cost, with retrieval and memory as one budget.
  • Define tool contracts and build evaluation practice: schemas, results, failure modes, and regression gates for every change.
  • Shape how Mimo reasons, asks, explains, and recovers, and decide which model runs where.

MimoLabs builds Mimo, an agentic system for making physical things, from design to production across 3D printing, CNC, and laser cutting. We are a small, VC-funded, experienced team based in Detroit and remote across the US, fast, adaptable, and honest.

Global

  • Review and evaluate multiple business presentation slide decks
  • Compare presentations to determine which exhibits higher professional quality
  • Provide detailed reasoning for your comparative assessments verbally

Terac is building the world's largest pool of vetted human experts for AI. Researchers, AI labs, and product teams use Terac to recruit, screen, and pay study participants across industries, languages, and skill sets.

$63–$120/hr
Global

  • Design and execute test cases to validate AI-generated outputs.
  • Review and annotate datasets to improve machine learning model quality.
  • Analyze model predictions and report findings to enhance performance.

Jobgether is an AI-powered job matching platform that connects candidates with roles from partner companies. It uses algorithmic shortlisting to ensure applications are reviewed quickly and fairly, and it emphasizes data privacy and transparent hiring processes.

Singapore

  • Evaluate and audit multilingual AI-generated responses for accuracy, helpfulness, and cultural relevance.
  • Review assigned batches of AI outputs against localized quality guidelines.
  • Score responses and rank multiple AI-generated answers based on defined quality criteria.

RWS is a language solutions company that provides AI training and linguistic quality evaluation services. They emphasize diversity, equity, and inclusion, offering flexible freelance opportunities for evaluators.

$30–$60/hr
Global

  • Design realistic marketing tasks with clear objectives, data, and deliverables.
  • Create reference solutions including campaign plans, market analyses, and performance reports.
  • Evaluate AI-generated output for accuracy, strategic reasoning, and execution quality.

Anyone AI is an AI company focused on developing and evaluating AI agents for real-world business tasks. They work with remote contractors to design marketing challenges and assess AI performance.

China

  • Evaluate AI-generated audio across naturalness, clarity, fidelity and overall quality.
  • Assign absolute scores and conduct pairwise comparisons in accordance with guidelines.
  • Write comments to explain any defects found in AI-generated audio.

RWS specializes in technology-enabled language services and AI training data. The company embraces diversity, equity and inclusion, and promotes equal opportunity for all employees.

Global

  • Review technical slide decks from your specific domain.
  • Rank presentations based on clarity, structure, and technical communication.
  • Provide clear feedback on what makes a specific presentation effective.

Terac is building the world's largest pool of vetted human experts for AI, enabling researchers and product teams to recruit, screen, and pay study participants. The company connects professionals across industries and skill sets for flexible remote work on AI training.

$286,000–$312,000/yr
US

  • Judge the AI's work by reviewing its output like a teammate's pull request.
  • Pair with the AI on your own task to flag where the model goes wrong.
  • Work on your own schedule with a fixed-pay contract earning $6,000/wk for 40hrs.

Handshake powers a platform connecting 25 million job seekers, 1 million+ employers, and 1,600 educational institutions, and has built a fast-growing AI data business working with frontier AI labs. They have grown to a ~$1B run rate, paying ~$60M to over 30,000 individuals monthly, with a culture focused on shaping careers in the AI economy.

Global

  • Design and ship features end to end across data model, API, background workers, and UI.
  • Build and improve agent and model evaluation: sandboxed trials, scoring, and trustworthy aggregation of thousands of runs.
  • Improve benchmark task quality checks, integrate new models and agents, and keep the system fast and reliable as usage grows.

Turing accelerates superintelligence by providing high-quality datasets, reinforcement learning environments, and research benchmarks to frontier AI labs, and also deploys agentic AI systems for Fortune 500 enterprises. Headquartered in San Francisco, it operates with startup speed and an AI-forward culture.

Global 7w PTO

  • Build and own evaluation infrastructure for Circle's AI agents.
  • Diagnose quality breakdowns across the agent pipeline.
  • Grow datasets and lead the AI Quality engineering team.

Circle builds an AI-powered all-in-one platform for digital businesses, uniting audiences with discussions, live streams, events, chat, courses, and payments. It's a fully remote team of about 270 people from 30+ countries, with an async culture and annual global offsites.

$80–$150/hr

  • Design evaluations of conceptual reasoning in AI safety, covering governance, alignment, macrostrategy, and economics.
  • Develop rubrics that separate genuine reasoning from plausible-sounding filler.
  • Iterate on tasks with frontier models and collaborate with a top research team.

Trajectory Labs is a startup automating AI safety research. We build safety and alignment evals, red-teaming programs, and RL environments for frontier AI labs. As a one-year-old company, we've been featured in major frontier lab safety reports and led high-impact red-teaming efforts.

$175,000–$195,000/yr
US Canada Unlimited PTO 12w maternity 12w paternity

  • Build evaluation frameworks for AI agents, including datasets and metrics for real-world task performance.
  • Design experiments to compare models and tools, identifying regressions and trade-offs.
  • Investigate agent failures and improve performance across models, harnesses, and tooling.

Gauntlet builds the financial systems of the future, operating across the onchain finance stack to offer vault products. They serve over $1.7B in client TVL across fintechs, protocols, exchanges, and capital allocators, combining traditional finance and crypto-native expertise.

Global

  • Work through complex mechanical engineering problems while narrating your reasoning process.
  • Evaluate AI-generated responses for accuracy, depth, and reasoning quality.
  • Write improved exemplars and structured feedback to enhance AI model performance.

Prolific is building the largest pool of quality human data for AI, used by over 35,000 developers and researchers. We provide a global platform for flexible, remote work to contribute to ethical AI development.

US

  • Own quality for non-deterministic, multi-agent AI systems, building the evaluation harness that underpins AI governance.
  • Verify grounding and faithfulness, building automated checks to stop hallucinations from reaching production.
  • Red-team for prompt injection, jailbreaks, and tool misuse, and prove multi-tenant data isolation.

SOFICO Intelligence uses AI to extract value from back-office systems, building agentic AI for financial, leasing, fleet, and mobility data. Backed by global ERP leader SOFICO, it's a remote-first team of curious, self-directed builders.

$20–$26/hr
Global

  • Complete side-by-side evaluations of AI-generated Bosnian text to assess naturalness and authenticity.
  • Rate how natural and idiomatic the writing sounds, identifying unnatural phrasing or cultural mismatches.
  • Perform quality control and contextual review to ensure tone and register match the cultural context.

Prolific builds the world's largest pool of quality human data for AI. Over 35,000 developers and organizations use Prolific to gather diverse data from paid participants.