Data Annotation Specialist

Responsibilities:

• Evaluate and rank model outputs: Complete preference and comparison tasks, assessing which responses best conform to project guidelines for accuracy, helpfulness, tone, and safety, and writing clear justifications for your judgments.
• Stress-test and break models: Probe models adversarially to surface failure modes, unsafe behavior, and capability gaps, and document reproducible cases that engineering and research teams can act on.
• Create datasets: Author high-quality prompts, responses, and exemplars to build training and evaluation datasets, following detailed specifications and editing machine-written or human-written outputs to standard.

Qualifications:

• 1+ years of experience in AI data annotation, LLM evaluation, content moderation, research, or a related analytical role, with exposure to quality assurance and/or preference ranking.
• Experience applying detailed guidelines to complex and often ambiguous content, with strong contextual and sociocultural judgment, sensitivity to nuance, tone, and register, and the ability to reason well in cases where there is no single correct answer.
• Comfort with ambiguity: a willingness to flag unclear or uncovered edge cases rather than guess, and to work productively on novel, experimental tasks whose definitions are still evolving.

Work environment:

• This is a part-time independent contractor position available within Canada only, requiring 16 hours per week at a CAD $30/hour contract rate.
• You will work remotely with your own device, and maintain control over how you complete your work, with the ability to work with multiple clients simultaneously.
• Continuous work is not guaranteed; engagements are project-based with potential fluctuations in workflow.