Clinical AI Evaluation Specialist

Role Focus:

• This role is at the intersection of behavioral healthcare, artificial intelligence, and product quality, focusing on defining standards for AI outputs.
• It involves enhancing evaluation and monitoring systems to ensure AI features meet clinical practice standards and governance policies.
• The specialist serves as a clinical subject matter expert, collaborating with product and engineering teams.

Key Responsibilities:

• Design evaluation frameworks, develop automated monitoring approaches, and conduct structured reviews of AI-generated content.
• Create and maintain reusable governance artifacts like system cards, monitoring playbooks, and risk assessment documentation.
• Participate in risk tiering assessments, support incident investigations, and contribute to AI literacy and training content.
• Stay current with developments in AI evaluation methodology, LLM behavior, and ABA best practices.

Qualifications:

• Must be a Board Certified Behavior Analyst (BCBA) in good standing.
• Requires 2+ years of experience in quality assurance, evaluation, or data analysis, preferably in healthcare settings.
• Demonstrated experience with large language models, generative AI, and prompt engineering is essential.
• Strong analytical skills and familiarity with HIPAA compliance, healthcare data standards, and ethical AI considerations are required.