Source Job

Global

  • Design, build, and maintain cybersecurity benchmarks for AI within evaluation infrastructure using Inspect.
  • Sandbox and harden evaluation environments to prevent AI escape and maintain security.
  • Integrate with AI providers to run benchmarks and collaborate with researchers for accurate, insightful outputs.

Software Engineering Red Teaming

20 jobs similar to Software Engineer, Cybersecurity Benchmarks

Jobs ranked by similarity.

US Unlimited PTO

  • Architect and scale agentic delivery platforms for application security assessments, threat modeling, and code reviews.
  • Serve as a security architect and AI strategy lead for client engagements and AI lab partnerships.
  • Drive secure-by-design AI systems aligned to OWASP, NIST, and MITRE frameworks while mentoring internal teams.

GuidePoint Security provides trusted cybersecurity expertise, solutions, and services that help organizations minimize risk. With over 1,300 employees, it serves more than 6,200 customers and maintains a collaborative culture driven by core values.

Americas

  • Own technical security controls for AI usage, including agent identity, tool access, guardrails, and logging.
  • Partner with AI Enablement, Engineering, DevOps, GRC, and Security to integrate security by design.
  • Build AI-powered security tooling and lead AI-specific incident response.

Alpaca is a US-headquartered global leader in agent-first brokerage infrastructure, serving financial institutions across 40 countries. With 400+ distributed team members and $400M in funding, we are a diverse, remote-first team opening financial services to everyone.

Global

  • Design and build automated evaluation pipelines for AI agents.
  • Develop testing frameworks for reasoning, multi-turn conversations, and adversarial scenarios.
  • Own the evaluation architecture and integrate observability tools in a fully remote team.

A company building production-grade AI agents and systems. They operate in a fully remote, globally distributed environment with a culture focused on innovation, autonomy, and continuous improvement.

US

  • Perform threat modeling and secure design reviews for new agent capabilities including tool use, memory systems, and multi-agent orchestration flows.
  • Research emerging attack surfaces relevant to AI agent platforms and translate findings into actionable mitigations.
  • Support enterprise customer security reviews and trust conversations related to agent security posture.

Emergence AI is a frontier agentic AI lab advancing safe, autonomous AI for enterprise environments. The team brings experience from leading AI research labs and tech companies, with offices in New York, California, Spain, and India.

$275,000–$300,000/yr
US Europe

  • Set technical direction for production agent systems, including orchestration, tool use, memory management, and failure recovery.
  • Build and operate shared foundations: evaluation frameworks, observability, execution controls, and model access with routing and failover.
  • Work with security and platform teams to ensure scoped permissions, data isolation, audit trails, and safe execution of tools and code.

Cobalt runs AI agents in production that test real customer applications, operating at the forefront of the pentesting industry. The company is rapidly expanding, offering a collaborative culture with mentorship, competitive compensation, and a focus on growth.

North America

  • Architect detection engineering and incident response strategy across cloud, identity, endpoints, and AI/ML pipelines.
  • Design AI-assisted SecOps workflows that automate triage, enrichment, and containment using LLMs and custom tooling.
  • Lead investigations, purple-team testing, and threat intelligence to defend against AI-driven threats.

Fullscript is a health technology company that empowers practitioners to deliver better care through clinical insights, analytics, education, and supplements. With over 125,000 practitioners and 10 million patients, it fosters a people-first, collaborative culture and flexible remote-friendly work.

Global Unlimited PTO

  • Design, develop, and deploy AI-powered tools, agents, and automation that streamline workflows across the organization.
  • Connect LLMs and AI applications with internal systems, APIs, and enterprise tools to enable intelligent operations.
  • Own the infrastructure, security, monitoring, and reliability needed to scale AI solutions in production.

Phantom is building a self-custody crypto wallet and trading app that provides access to global markets, including perpetuals, prediction markets, tokenized assets, stablecoins and memes. The company is around 180 people, fully remote, and backed by a $150M Series C from a16z, Sequoia Capital and Paradigm.

  • Design and build offensive cybersecurity reinforcement learning environments.
  • Harden existing benchmarks to withstand advanced AI interactions.
  • Develop rigorous evaluation criteria for frontier AI models.

Terac is building the world's largest pool of vetted human experts for AI. Researchers, AI labs, and product teams use Terac to recruit, screen, and pay study participants across industries, languages, and skill sets.

Global

  • Build and maintain shared agent tools and capabilities in production.
  • Configure prompt agents and customize tools for each use case.
  • Deploy agents, monitor fleet health, and optimize in production.

EQT builds a fleet of AI agents that work alongside our people. The company emphasizes transparent leadership, connected impact, and supported growth, fostering a collaborative culture where every voice is heard.

India

  • Deploy and operationalize AI security solutions in customer environments, translating requirements into scalable architectures.
  • Design integrations across AI platforms, model registries, and security tooling to secure AI systems.
  • Lead cross-functional collaboration with AI engineering, security, and data teams to drive production-ready implementations.

AHEAD builds platforms for digital business by integrating cloud infrastructure, automation, and software delivery to help enterprises with digital transformation. They promote an inclusive and diverse culture with a multi-million-dollar lab, emphasizing cross-department training and development.

US

  • You'll lead and develop the Webapp and AI Attack engineering teams, setting direction, goals, and expectations.
  • You'll own the strategy and execution of the portfolio, partnering with product, platform, and customer-facing teams.
  • You'll guide capabilities from research through production, including web application testing and AI/LLM security assessments.

Horizon3 is a fast-growing, remote cybersecurity company that builds NodeZero, an autonomous pentesting platform for finding exploitable attack vectors. They are a fusion of former special ops cyber operators and engineers, committed to a culture of respect, collaboration, ownership, and results.

Global

  • Lead technical discovery and turn ambiguous cyber needs into a tractable plan with clear architecture and acceptance criteria.
  • Build and integrate AI-enabled security workflows using production software, APIs, telemetry, and security systems.
  • Deploy, operate, and evaluate AI workflows in customer environments, ensuring security, maintainability, and handover.

Eliza is a technology services company and Advanced-tier OpenAI partner that helps organizations build and deploy AI solutions, including generative AI, custom LLM integrations, predictive analytics, and intelligent automation. The company has a collaborative, mission-driven team focused on real-world AI impact and values adaptability, ownership, and hands-on client collaboration.

$80–$150/hr

  • Design evaluations of conceptual reasoning in AI safety, covering governance, alignment, macrostrategy, and economics.
  • Develop rubrics that separate genuine reasoning from plausible-sounding filler.
  • Iterate on tasks with frontier models and collaborate with a top research team.

Trajectory Labs is a startup automating AI safety research. We build safety and alignment evals, red-teaming programs, and RL environments for frontier AI labs. As a one-year-old company, we've been featured in major frontier lab safety reports and led high-impact red-teaming efforts.

$75,500–$96,000/yr
US 4w PTO

  • Build AI agents and LLM-assisted workflows to triage, enrich, and resolve security alerts.
  • Develop and maintain security data pipelines that collect, normalize, and route telemetry.
  • Collaborate with Cyber Operations to automate manual steps and document decision paths.

HealthEquity empowers healthcare consumers by providing HSAs and related financial solutions. The company fosters a collaborative, purpose-driven culture and values authenticity and innovation.

India

  • Deliver end-to-end threat intelligence enablement and technical onboarding, optimizing customer digital footprints and alert rules for noise-free monitoring.
  • Develop custom automation scripts, AI agents, and SOAR playbooks to streamline security workflows and enhance platform adoption.
  • Drive customer retention through proactive engagement, churn risk analysis, and adaptive threat reporting.

SOCRadar provides Extended Threat Intelligence (XTI) solutions that help organizations proactively secure their attack surfaces and gain visibility into surface, deep, and dark web threats. The company is focused on innovation, global expansion, and delivering automation-enabled security to enterprises worldwide.

Europe

  • Lead and own security across the company, hardening cloud infrastructure and managing vulnerabilities.
  • Conduct hands-on penetration testing, code review, and incident response for AI-driven systems.
  • Build automated security controls into existing workflows without slowing delivery.

Zencoder builds and orchestrates AI agents that ship real work, from code to operations. The company keeps a small, highly capable team with a remote-first culture, prioritizing talent and autonomy.

US Unlimited PTO 16w maternity 16w paternity

  • Plan, build, and improve background-agent infrastructure that enables reliable, trackable, and repeatable automated engineering work.
  • Develop systems for evaluating agent performance using task outcomes, quality measures, human intervention, cost, and other signals.
  • Improve how engineers discover, understand, and use shared AI tools and workflows, and create clear guidance and documentation.

Vanta helps businesses earn and prove trust through automated security monitoring and a leading Trust Management Platform. Founded in 2018, the company has a kind, inclusive, and talented team, and thousands of companies rely on Vanta for real-time security and compliance.

AI Engineer

Elicit
$185,000–$320,000/yr
US 4w PTO

  • Build evaluations to measure and improve AI reasoning in complex, high-stakes decisions.
  • Collaborate with engineers, product, and domain experts to turn observed failures into testable improvements.
  • Own the quality of shipped systems, from code reviews to monitoring and integration.

Elicit builds AI systems that help research and business teams make evidence-based decisions on high-stakes questions. They are a remote-friendly, collaborative team that values rigorous thinking, experimentation, and in-person retreats.

Global 18w maternity 18w paternity

  • Set UpGuard's automation and AI strategy, define OKRs, and own the roadmap for the internal AI co-worker ug-buddy.
  • Lead and coach a team of AI and automation engineers, run sprint delivery, and ensure production systems are secure and reliable.
  • Drive cross-functional automations, set platform standards, and work with departments to identify efficiency opportunities.

UpGuard is a cyber risk management company that builds an AI-driven platform to replace manual security bottlenecks. It is a globally recognized, well-funded employer with a highly engaged team and a great work culture.