Source Job

United States

  • Help shape how AI handles medicine by applying clinical judgment to AI evaluation tasks.
  • Perform factuality checks, assess LLM-generated medical content for accuracy and bias.
  • Enjoy flexible, remote freelance work with competitive hourly rates and no minimum hours.

Clinical Judgment Medical Knowledge Attention To Detail Fact-Checking

20 jobs similar to New Resident Physician - AI Training - Improving AI Trust and Safety

Jobs ranked by similarity.

United States

  • Use your clinical judgment to improve healthcare AI trust and safety.
  • Perform factuality checking on LLM generated materials for accuracy.
  • Evaluate AI writings for accuracy, relevance, and bias.

Prolific is an AI data platform that builds a pool of quality human data for AI development. They have over 35,000 users and emphasize ethically sourced data.

US

  • Complete expert registration and medical-credential verification through Prolific's onboarding process.
  • If invited, complete a training or qualification assessment to establish suitability for the project.
  • Apply medical expertise to paid research and AI-related tasks as they become available, following confidentiality requirements.

Prolific is building the world's largest pool of high-quality human data for AI research. Over 35,000 AI developers and researchers use Prolific, which emphasizes ethically sourced data and flexible, remote participation.

US

  • Complete expert registration and medical-credential verification through Prolific’s onboarding process.
  • Apply your laborist or obstetric hospitalist expertise to paid AI research and task-based projects.
  • Participate remotely on a freelance basis, choosing tasks that fit your schedule with pay from $216/hour.

Prolific builds a platform for collecting high-quality human data, connecting AI developers and researchers with paid study participants. With over 35,000 users, it focuses on ethically sourced behavioral data to improve AI systems.

US

  • Complete expert registration and medical-credential verification through Prolific's onboarding process.
  • Complete a training or qualification assessment to establish suitability for AI-related tasks.
  • Apply your per diem physician experience to paid, task-based research projects with rates starting at $216/hour.

Prolific builds the world's largest pool of quality human data, connecting over 35,000 AI developers and researchers with paid study participants. The platform provides ethically sourced, high-quality behavioral data to help train more accurate and aligned AI systems.

$100–$175/hr
Global

  • Review de-identified patient records, clinical notes, and model responses; label them against clear guidelines.
  • Write and refine grading rubrics and reference answers for clinical reasoning tasks.
  • Grade AI model outputs for accuracy, safety, and clinical usefulness, and flag ambiguous cases.

GenMD is a Stanford spin-off building AI and data infrastructure for healthcare. We're a small, well-funded, revenue-generating team fresh out of stealth, moving fast and focused on building something fundamental.

US

  • You'll annotate medical images using your professional expertise.
  • You'll review and verify AI-generated interpretations, correcting inaccuracies.
  • You'll evaluate model outputs and provide expert feedback for clinical accuracy.

Prolific is building the biggest pool of quality human data in the world, connecting over 35,000 AI developers and researchers with paid study participants. We're a remote-first platform that values diverse human perspectives and offers flexible work to help shape the future of AI.

United States

  • Evaluate long-form AI-generated research outputs against detailed rubrics and guidelines.
  • Check claims, tables, and citations for accuracy, distinguishing primary sources from low-quality pages.
  • Write short, specific rationales pointing to concrete evidence.

Welo Data provides AI services and data solutions. They are a growing company that values quality and consistency in AI evaluation.

Global

  • Review de-identified clinical cases and pharmaceutical product images.
  • Develop scoring rubrics and provide written expert evaluations.
  • Assess accuracy of medical data interpretations for AI benchmarking.

Terac is building the world's largest pool of vetted human experts for AI. Researchers, AI labs, and product teams use Terac to recruit, screen, and pay study participants across industries, languages, and skill sets.

Brazil

  • Evaluate AI-generated responses for scientific accuracy and logical consistency.
  • Test AI models with theoretical biology questions and realistic laboratory scenarios.
  • Identify and document model failure modes, recommending improvements to prompts and evaluation methods.

A partner company of Jobgether focused on improving AI language models through expert scientific training and evaluation. The organization works with freelance specialists to enhance AI reasoning and reliability in biological domains.

Global

  • Review de-identified medical images and associated clinical cases through a secure remote platform.
  • Provide structured expert reads and complete standardized scoring rubrics for diagnostic benchmarking.
  • Articulate diagnostic reasoning and clinical confidence while working asynchronously at your own pace.

Terac is building the world's largest pool of vetted human experts for AI, connecting researchers with study participants across industries and skills. The company enables flexible, paid expert work through a secure remote platform.

$80–$150/hr

  • Design evaluations of conceptual reasoning in AI safety, covering governance, alignment, macrostrategy, and economics.
  • Develop rubrics that separate genuine reasoning from plausible-sounding filler.
  • Iterate on tasks with frontier models and collaborate with a top research team.

Trajectory Labs is a startup automating AI safety research. We build safety and alignment evals, red-teaming programs, and RL environments for frontier AI labs. As a one-year-old company, we've been featured in major frontier lab safety reports and led high-impact red-teaming efforts.

$30–$60/hr

  • Write questions and tasks that isolate and test specific model capabilities.
  • Develop rubrics that define strong, partial, and poor answers.
  • Grade model outputs against rubrics and explain scoring decisions.

$17–$17/hr
Global

  • Submit a real question you would actually ask an AI, typed naturally in your own words.
  • Tell why you needed to know and list two or three things a good answer must cover.
  • Earn $0.55 per accepted submission; no interview, no minimum hours, fully remote.

Kled is building a consumer app that pays people to upload their real photos, videos, and documents, then licenses that rights-aware data to AI labs and enterprises. Since 2026, it has scaled to 600,000+ active contributors, processes 5M uploads per day, and raised $15M+ from top investors.

  • Review de-identified medical images across multiple modalities
  • Provide expert diagnostic reads and clinical interpretations
  • Apply structured scoring rubrics to clinical cases

Terac is building the world's largest pool of vetted human experts for AI. Researchers, AI labs, and product teams use Terac to recruit, screen, and pay study participants across industries, languages, and skill sets.

Global

  • Evaluate AI-generated chemistry responses for factual accuracy and safety.
  • Fact-check chemical reactions, formulas, and synthetic route viability.
  • Annotate molecular data and correct errors in SMILES strings and representations.

We are building the biggest pool of quality human data in the world. Over 35,000 AI developers, researchers, and organizations use Pralific to gather ethical data from paid study participants.

$140,000–$180,000/yr
Global 2w PTO

  • Design tasks for AI agents on medical imaging, including writing questions and defining pass criteria.
  • Review agent performance, identifying missed or falsely reported findings.
  • Work across MRI, CT, ultrasound, and radiography to build benchmarks.

Latch builds rigorous scientific benchmarks for AI agents across biology. We are a small, transparent team that values collaboration and continuous improvement.

United States

  • Evaluate and provide feedback on search engine results, text, audio, images, and videos to enhance AI data quality.
  • Conduct online research to assess relevance and significance of content, ensuring accuracy and quality standards.
  • Work flexible part-time hours (10-20 per week) remotely, contributing to the improvement of AI-powered search experiences.

RWS is a global leader in language services and technology, providing translation, localization, and AI training data solutions. The TrainAI division focuses on improving AI models through human feedback, fostering a flexible and inclusive culture for remote workers.

UK US

  • Review anonymised conversations with Ash and judge, as a clinician, whether it handles key moments well.
  • Explain your reasoning on what Ash should do, what could go wrong, and why.
  • Work alongside experienced clinicians without contacting users, helping shape what great looks like.

Slingshot AI is the team behind Ash, the first AI designed for mental health, making support more accessible. It's a well-funded Series A company with $93M raised, offices in NYC and London, and a culture of autonomy, flexibility, and care.

US

  • Evaluate AI-generated clinical interactions for accuracy, safety, and appropriateness.
  • Annotate and score AI responses against evidence-based protocols and structured rubrics.
  • Provide detailed written feedback and participate in calibration sessions to enhance AI reliability.

The company focuses on improving AI-powered clinical solutions by leveraging clinical expertise. It is an early-stage partner organization offering flexible, remote contract opportunities for healthcare professionals.

Global

  • Work through complex mechanical engineering problems while narrating your reasoning process.
  • Evaluate AI-generated responses for accuracy, depth, and reasoning quality.
  • Write improved exemplars and structured feedback to enhance AI model performance.

Prolific is building the largest pool of quality human data for AI, used by over 35,000 developers and researchers. We provide a global platform for flexible, remote work to contribute to ethical AI development.