Source Job

US Unlimited PTO

  • Lead the engineering strategy and execution for evaluations of AI agents, owning the core evaluation platform.
  • Design scalable evaluation infrastructure, APIs, workflows, and production systems across software categories.
  • Mentor and develop a team of engineers as the technical authority on agentic evaluation.

Python TypeScript Agentic AI Engineering Management

20 jobs similar to Software Engineering Director, Agentic Evaluations

Jobs ranked by similarity.

$336,000–$400,000/yr
US

  • Lead the AI Platform team to build the technical rails for AI-native operations, including LLM gateway, agent framework, and evaluation infrastructure.
  • Drive the Velocity program to achieve true autonomy by optimizing product delivery cycle time with AI-written code and coverage gates.
  • Grow the platform org through a Director and principal engineers, holding the line on a single shared platform across all domains.

Engine is the all-in-one travel and spend management platform trusted by 38K+ businesses and 1.8M+ travelers. With 1,000+ employees, we are one of the fastest-growing companies in travel tech, recognized as a Best Place to Work and a leader in AI innovation.

$200,000–$400,000/yr
Global 2w PTO

  • Lead the development of the AIOS Agent SDK, forming the foundation for world-class agents across the company.
  • Own agent architecture, model strategy, evals, and production reliability for AI systems handling $100k daily revenue.
  • Build and scale the applied AI team, setting technical direction and ensuring safe, compliant agent operations.

AIOS is building the world’s first full-stack AI doctor, serving 150k patients monthly via Bolt Pharmacy in the UK. They are a profitable, fast-growing startup with $350M ARR and a founder-led culture of intense execution.

US

  • Lead the Agent Tools, Agent Observability, and Runner Execution teams across frontend, backend, and AI engineering.
  • Partner with product and UX to turn ambiguous problems into shipped milestones and make prioritization tradeoffs explicit.
  • Contribute substantively to technical design for running agents safely and effectively at scale.

GitLab is the intelligent orchestration platform for DevSecOps, enabling organizations to increase developer productivity and accelerate digital transformation. With over 50 million registered users and more than 50% of the Fortune 100 trusting them, GitLab fosters a high-performance culture driven by values, continuous knowledge exchange, and AI integration.

North America

  • Design and build production-grade AI agentic systems for web and chat experiences.
  • Architect agent workflows involving reasoning, tool use, retrieval, guardrails, and monitoring.
  • Build LLM evaluation systems and diagnose agent performance issues across prompts and tools.

Netomi is an agentic AI platform for enterprise customer experience, enabling automation for global brands. Backed by Y Combinator and Index Ventures, the company drives efficiency and higher quality experiences.

Europe

  • Lead a team of AI engineers, setting technical direction and quality standards while staying hands-on in code.
  • Own the architecture and end-to-end delivery of AI systems using LLMs, RAG, and agentic patterns.
  • Drive evaluation, monitoring, and continuous improvement loops to ensure production-grade AI capabilities.

Finom is a European tech startup building an all-in-one financial B2B platform for entrepreneurs, integrating banking, accounting, and invoicing. They are a well-funded company with over $346 million in total funding, maintaining a startup culture that values innovation, swift implementation, and employee impact.

$221,200–$387,100/yr
North America

  • Lead the architecture and direction of a group of Forward Deployed Engineers building autonomous intelligence systems.
  • Translate customer use cases into precise specifications and direct AI agents to generate production code.
  • Engineer context for reliable agent output and verify code correctness, security, and maintainability.

ServiceNow provides an AI platform for business reinvention, automating workflows to free people from busywork. They serve 85% of the Fortune 500 and foster an AI-native culture where technology and talent are unstoppable together.

$169,100–$235,700/yr
US

  • Lead and grow the AI Transformation engineering team, owning delivery, quality, and operational health.
  • Guide architecture of agentic AI systems and set engineering standards for security and compliance.
  • Partner with business units to align engineering delivery with business outcomes and track KPIs.

GoodLeap is a technology company providing financing and software for sustainable home solutions like solar and energy-efficient upgrades. With over 1 million homeowners served and $30 billion in financing since 2018, they support a collaborative culture and are dedicated to positive environmental impact through their nonprofit, GivePower.

$88,000–$180,000/yr
US Unlimited PTO

  • Direct the agent array on production workstreams by decomposing problems into tasks and integrating agent output into shipped software.
  • Review agent-generated pull requests at volume and depth, identifying correctness, security, and accessibility defects.
  • Author evaluation suites that make quality measurable using eval-driven development and own end-to-end quality within a FedRAMP-authorized environment.

Granicus provides cloud-based solutions for government communications, website design, meeting management, and records management, serving over 5,500 agencies and 300 million citizens. With a globally distributed team and a culture of transparency and inclusion, Granicus has been recognized on the GovTech 100 list for the past 5 years.

UK

  • Build tooling for capturing and processing data from agents and humans at significant scale.
  • Solve hard problems around compute, orchestration, scaling, security, and reliability.
  • Help develop approaches for training, benchmarking, and evaluating AI agents.

Prolific builds human data infrastructure for AI development, connecting researchers with a global pool of participants to collect high-quality, ethically sourced behavioral data. They are a mission-driven company at the forefront of AI innovation, with a remote culture and a focus on impactful work.

$145,000–$250,000/yr
US Unlimited PTO

  • Design, implement, and operate benchmark execution and evaluation harnesses for AI models and agentic workflows.
  • Develop evaluation methodologies that combine automated metrics with structured human subject matter expert judgment.
  • Produce defensible evaluation reports comparing candidate capabilities with current mission workflows, including documented limitations and failure modes.

OpenTeams helps enterprises and governments build AI they control, govern, and evolve themselves. Founded by NumPy and SciPy creator Travis Oliphant, the company is built by people with deep roots in the open-source ecosystem.

UK 4w PTO

  • Design, build, and operate production-grade agentic systems for developers.
  • Develop core agent infrastructure including orchestration, tool execution, and evaluation.
  • Own projects end to end from opportunity identification to launch and iteration.

Our partner is a fast-growing Series C technology company that builds AI-powered developer tools. The organization is fully remote with an autonomous culture focused on engineering ownership and collaboration.

Global

  • Develop and deploy cutting-edge AI/ML solutions to enhance the platform and improve student learning experiences.
  • Design, develop, and optimize LLM-powered agentic systems and APIs for real product use cases.
  • Collaborate with senior engineers and contribute to evaluation frameworks and MLOps pipelines.

Interview Kickstart specializes in interview preparation and career transitions into high-demand tech fields like AI, ML, and Data Science. Over 17,000 tech professionals have been guided by current and former hiring managers to land coveted positions at companies like Google and Amazon.

$250,000–$300,000/yr
US Canada Unlimited PTO

  • Own the build out of new agents, skills, and platform capability for teams across TLDR.
  • Build and deploy agents end to end, from design through implementation, evals, and rollout to internal users.
  • Partner with stakeholders across sales, editorial, and people ops to find where an LLM belongs in their process.

TLDR runs the largest network of tech newsletters in the world, with over 8 million subscribers covering startups, software engineering, AI, and more. Our 31-person full-time team is bootstrapped, profitable, and on track for $35M in revenue this year, with a culture of owning functions rather than slices.

Europe

  • Architect and build production AI-agent systems like AIDA and NOVA, shaping how Yuno applies LLMs and agentic AI to payment problems.
  • Design conversational and multilingual agents that hold real customer conversations across channels, optimized for reliability and low latency.
  • Define evaluation, safety, and guardrails for agentic systems in a compliance-sensitive, multi-market environment.

Yuno is the AI-native operating system of global commerce, powering financial infrastructure for enterprise merchants, banks, and wallets. It connects over 1,000 payment methods across 190+ countries, trusted by global brands like McDonald's and Rappi, with a remote-first culture.

Global

  • Design and implement autonomous agents and multi-agent orchestration systems for complex, open-ended tasks.\n- Build and optimize production-grade LLM applications with a focus on reliability, observability, and low-latency performance.\n- Architect advanced RAG pipelines and vector database strategies to provide agents with accurate, real-time context.

Jobgether is an AI-powered job matching platform that connects candidates with hiring companies through objective evaluation. The company operates with a distributed, global-first team culture, emphasizing fairness and innovation in recruitment.

$185,000–$215,000/yr
US Unlimited PTO 20w maternity 16w paternity

  • Build the planning and execution systems that let Neo safely perform multi-step infrastructure operations across AWS, Azure, GCP, Kubernetes, and more.
  • Design tool interfaces and context systems to give the agent semantic understanding of cloud resources and access to engineering workflows.
  • Create feedback loops, self-correction mechanisms, and evaluation frameworks to ensure reliable autonomous operation.

Pulumi is reimagining how teams build cloud software, enabling developers and infrastructure experts to work better together through a unique combination of programming languages, tools, and systems innovation. With a global community of over 350,000 members and serving more than 3,250 customers, Pulumi is a remote-first company that fosters a culture of innovation and inclusion.

North America 3w PTO

  • Develop, configure, deploy, and optimize AI agents using Cresta's AI platform and tools.
  • Build AI agent integrations with external systems such as APIs, databases, and CRMs to ensure seamless workflow integration.
  • Collaborate with customers and internal stakeholders to gather technical requirements and translate business needs into AI agent solutions.

Cresta builds a unified AI platform that combines conversational AI agents, real-time human agent augmentation, and conversation intelligence to drive revenue and efficiency for customer experience channels. Born from Stanford AI Lab, the company has raised over $270 million from top investors and is led by AI industry veterans.

Global 4w PTO

  • Build and ship agentic features for thousands of developers, including orchestration, tool execution, and context management.
  • Design and implement the foundation for advanced AI agents that handle billions of dollars in purchase volume.
  • Collaborate with product and engineering teams to scope, prioritize, and drive rollout of new features.

RevenueCat provides infrastructure and tools for app businesses to manage subscriptions and monetization. They are a remote-first team of 150+ people across 25+ countries, guided by values like Customer Obsession, Always Be Shipping, Own It, and Balance.

Singapore

  • Advise enterprise customers on agent architecture and evaluation strategies.
  • Build and co-build production agents across the full Agent Development Lifecycle.
  • Embed within customer teams to ship agent systems directly as a deployed engineer.

LangChain makes intelligent agents ubiquitous by providing open-source frameworks and a platform for building, evaluating, and deploying AI agents. With $125M Series B funding and a growing team, we serve 35% of the Fortune 500 with a culture of impact and collaboration.

Europe

  • Lead a remote team of engineers in designing and shipping production-grade agentic AI systems for enterprise clients.
  • Own solution design and system architecture across agents, tools, data, evaluations, and controls.
  • Personally implement critical components and guide client stakeholders through difficult technical decisions.

EggAI is an enterprise-focused generative AI company that helps large organizations move AI solutions from prototyping to production. Based in Munich, Germany, the company builds safe, reliable, and scalable AI systems for enterprise clients.