New Analyst - LLM/Prompt Evaluation (3 Month Fellowship)

Blue Rose Research is seeking an analyst to play a vital role in maintaining and improving the effectiveness, accuracy, and fairness of our Large Language Model-powered tools and analyses. As we increasingly leverage LLMs for tasks ranging from feature extraction in political videos and powering chatbots, to generating persuasive scripts and scoring content effectiveness, ensuring the quality, reliability, and responsible deployment of these systems is paramount. This role involves significant hands-on quality control, evaluation design, proactive red teaming, bias analysis, and performance measurement to ensure our LLM outputs meet high standards and drive real-world impact during the fellowship period. You will own the evaluation lifecycle for our LLM applications by designing, implementing, and managing evaluation frameworks to systematically measure performance, accuracy, and reliability across diverse tasks (e.g., video analysis, summarization, chatbot outputs). Conduct rigorous quality control and analysis by meticulously reviewing LLM outputs, performing QC, analyzing results using SQL, identifying trends/weaknesses, and report findings clearly. You will proactively enhance LLM safety and fairness by executing red teaming analyses to uncover vulnerabilities and failure modes; analyze outputs for biases and contribute to mitigation efforts. Improve LLM effectiveness through iteration by collaborating with the end users of our LLM products to understand their needs and refine prompts to enhance output quality, safety, and utility. You will also document and communicate findings by maintaining clear records of processes and results, effectively communicating insights, including potentially sensitive ones, to stakeholders.