Source Job

• Design adversarial prompts to test AI models for offensive security capabilities.
• Evaluate model-generated code for functional correctness and real-world exploitability.
• Test model behavior across cybersecurity categories like malware, exploitation, and social engineering.

Handshake is a career platform that helps everyone find a path to a great career. They support 25 million job seekers, over 1 million employers, and 1,600 educational institutions.

• Review and advise on evaluation criteria and scoring rubrics for AI-generated outputs.
• Create, edit, and validate high-quality benchmark tasks and reference data for AI training.
• Analyze model failures, including hallucinations and flawed reasoning, providing expert explanations.

The company specializes in AI evaluation and training, helping define standards for next-generation AI systems. They operate as a partner company that manages applications and next steps, valuing autonomy and expertise in their consultants.

• Evaluate user requests and AI model responses involving violence, weapons, threats, and dark fiction within the full conversation context.
• Distinguish fictional, educational, historical, and defensive violence from requests that seek real-world uplift or express real intent to harm.
• Assess whether a model's response gives meaningful real-world capability, and write concise rationales citing policy language and conversation details.

Handshake is a career platform founded on the belief that everyone deserves a path to a great career. Today, it powers 25 million job seekers, 1 million+ employers, and 1,600 educational institutions, with a rapidly growing AI data business that has reached a ~$1B run rate.

• Lead the development and optimization of machine learning models to detect and prevent AI security risks like prompt injection and jailbreaks.
• Build reproducible training and evaluation pipelines on Reddit's ML platform, partnering with platform engineers to improve performance and reliability.
• Set the technical vision and multi-quarter modeling roadmap, mentoring engineers and establishing best practices for responsible ML development.

Reddit is a community of communities, built on shared interests and authentic conversations, with 100,000+ active communities and 130 million daily active visitors. It is one of the internet's largest sources of information, fostering a culture of openness and trust.

• Translate a complex workflow into a demanding AI prompt designed to expose model limitations.
• Test your prompt in ChatGPT, refine it until the AI fails, and write a grading rubric for others to use.
• Submit your prompt, failure notes, rubric, and a screen recording of your thought process.

Terac builds the world's largest pool of vetted human experts for AI research and evaluation. They are a growing platform used by AI labs and researchers to recruit, screen, and pay study participants globally.

• You will design the architecture for specialized subagents operating within live customer conversations, including technical QA, product expertise, and objection handling.
• You will build routing and delegation systems that determine when to answer directly, invoke a subagent, or escalate to a human.
• You will master the dialogue platform, train AI agents via prompting and fine-tuning, and document workflows to educate the team.

1mind builds autonomous customer experience software that deploys AI-powered 'Superhumans' to engage, demo, onboard, and support customers across the entire buying journey. The company offers a remote-first, fast-moving culture with ownership, autonomy, and impact from day one.

• Own research projects end-to-end: identify important questions, formulate hypotheses, design experiments, analyze results, and publish.
• Develop rigorous evaluations of misalignment and loss-of-control risks, including evaluation awareness, sandbagging, and dishonesty.
• Study behaviors difficult to observe directly, such as long-horizon failure modes and cases where models may conceal relevant behavior.

Neo Research is an independent AI safety research organization based in Singapore that studies frontier risks in increasingly capable AI systems, with a focus on open-weight models. The team is small, offering substantial freedom to pursue important research questions and a collaborative environment with strong engineering support and external relationships.

• Lead Affirm's enterprise AI security review process, evaluating architecture, data flows, and design of AI tools and agentic systems.
• Threat model AI/LLM systems for risks like prompt injection, insecure output handling, and data poisoning, and drive remediation.
• Build security guardrails, tooling, and policy-as-code to automate AI security and support cross-functional initiatives.

Affirm is a financial technology company that offers clear, predictable point-of-sale installment loans with no hidden fees. The company is remote-first and values transparency, care, and flexibility, with a focus on building a diverse and inclusive team.

• Design and implement layered AI guardrails and sandboxing to constrain AI behavior and secure enterprise workflows.
• Partner with users to bake secure-by-default patterns into AI-assisted workflows and maintain an inventory of AI-to-service connections.
• Continuously test guardrails through red-teaming and evaluate new AI tools to find secure ways to enable adoption.

Waabi, founded by AI visionary Raquel Urtasun, is the leader in Physical AI, developing autonomous transportation technology for commercial trucks and robotaxis. Backed by world leaders in AI and automotive, the company has offices in Toronto, San Francisco, Dallas, and Pittsburgh and is growing quickly with a diverse, innovative team.

• Evaluate user requests and AI model responses against detailed customer policies with precise reasoning.
• Distinguish subtle differences in context and intent to classify ambiguous cases accurately.
• Participate in calibration discussions and contribute to improving evaluation frameworks.

Handshake powers a platform connecting 25 million job seekers with employers and educational institutions. Through Handshake AI, they provide data to frontier AI labs, having grown to a ~$1B run rate and paying over 30K individuals monthly.

• Evaluate AI systems at a scale only possible by combining thousands of vetted experts with model graders.
• Innovate at the frontier of QA by shaping industry standards for validating agentic AI and large language models.
• Collaborate with global market leaders to architect AI quality blueprints and drive high-impact consultative visibility.

Testlio provides a fully managed crowdsourced testing platform powered by proprietary intelligence technology, LeoCore. They are a female-founded, fully remote company with an inclusive culture, half of their team identifying as women, and are growing profitably.

• Support participants in virtual HR workshops by diagnosing and improving generative AI outputs.
• Provide tool-agnostic guidance to help attendees refine prompts and achieve practical results.
• Ensure responsible AI use and escalate issues as needed while fostering productive learning.

This company is a leading technology firm specializing in internet-related services and products, including search, cloud computing, and AI. It is a large global organization with a culture of innovation and collaboration.

• Conduct manual penetration tests against core systems and AI systems, and build AI-assisted tooling to extend testing coverage.
• Help define how we pentest AI, including LLM applications, agents, and agent-generated code.
• Triage findings from SAST tools, fix vulnerabilities, and tune rules to reduce false positives.

Forward Financing is a fintech company that unlocks capital for small businesses across America. Since 2012, they have provided over $4.8 billion in funding to more than 92,000 small businesses and are recognized as a Best Place to Work.