LLM Quality Analyst

Work with the Trust, Safety and Quality team to support the quality and reliability of MagicSchool's AI outputs. This is an excellent early career role for individuals interested in exploring career paths like product management or data analysis. Work closely with the Senior LLM Quality Analyst, handling feedback intake, ground truthing LLM responses, and helping maintain evaluation test suites. Manage LLM Feedback Intake and Triage user feedback on LLM outputs, identify patterns in quality issues, and act as human-in-the-loop for quality checks. Manage and develop test cases by reviewing and labeling LLM-generated responses, generating ground truth datasets, and supporting evaluator and judge development. Build and Maintain Evaluation Test Suites for regular evaluation of LLM performance. Support Evaluation Dashboards by assisting in building dashboards to visualize and monitor evaluator outputs.