Similar Jobs
See allBrazilian Portuguese - AI Model Rater
Partner Company
Brazil
English
AI Evaluation
Attention To Detail
QA (Quality Assurance) Pleno/Sênior
Confidential
Brazil
Software Testing
Test Automation
Playwright
Senior Quality Engineer (AI)
Jobgether
Canada
Node.js
MongoDB
LLM Evaluation
Applied AI Engineer
HiPeople
Global
Go
AI/ML
LLMs
Senior QA Automation Engineer
College Board
US
Cucumber
Jest
AWS
Role Overview:
- This position is listed on behalf of a partner company, who manages all applications and next steps.
- Our partner is looking for an AI SDET based in Brazil.
- This role sits at the intersection of software quality engineering, artificial intelligence, and regulatory assurance.
Key Responsibilities:
- Own and manage the AI evaluation harness, including the creation, maintenance, and continuous improvement of automated evaluation workflows.
- Develop and maintain 'Golden Path' scenarios representing expected AI behavior and critical user journeys.
- Design adversarial and edge-case test scenarios to identify weaknesses in AI agents and expose failures that conventional testing may overlook.
Requirements:
- Advanced proficiency in Python 3.11+, Pytest, and modern automated software testing frameworks.
- Hands-on experience with AI evaluation platforms and methodologies, including Langfuse, RAGAS, or comparable tools.
- Practical knowledge of LLM-as-a-judge evaluation patterns and approaches for assessing non-deterministic AI outputs.
Jobgether
Jobgether is an AI-powered job matching platform that connects candidates with hiring companies. It uses AI to review applications and share top-fitting candidates with employers, focusing on efficient and fair recruitment.