Source Job

US

  • Design and iterate on agent behavior across real, live workflows, including long-horizon, multi-turn agentic tasks.
  • Build and run evaluations against production conditions to measure performance, regressions, and edge cases.
  • Partner with Product and Engineering to ensure agents are steerable, trustworthy, and ready for scale.

Machine Learning LLM RAG AWS Evaluation Frameworks

20 jobs similar to Staff Applied AI Engineer, Product & Agent Performance

Jobs ranked by similarity.

$139,000–$202,000/yr
Canada US

  • Design and build internal AI services that help sellers create tailored sales assets and respond to deal signals.
  • Integrate AI agents with platforms like Salesforce, Snowflake, and Slack for permission-aware tool use.
  • Evaluate and improve agent behavior through user feedback, quality signals, and production monitoring.

1Password builds a secure digital future with unified access management and enterprise password management, trusted by over 180,000 businesses. They are a remote-first company with a culture of collaboration and innovation, and have surpassed $400M in ARR, earning a spot on the Forbes Cloud 100.

Europe

  • Design and build reusable agentic infrastructure, including memory, context management, tool-calling, orchestration, and feedback loops.
  • Establish evaluation and observability frameworks to understand agent performance and ensure reliable production systems.
  • Drive operational automation by bridging complex workflows with engineering and influencing architectural patterns.

Dwelly builds an AI operating system for residential lettings, automating workflows for landlords and tenants. Managing over 15,000 properties and $470M GMV, it's a fast-growing, product-focused team backed by $263M.

AI Engineer

Ditto
Global Unlimited PTO

  • Build the agent execution runtime: Design and ship the layer that compiles a brief into a running team of agents.
  • Design the tool surface agents work through: Replace "here is a shell, please be careful" with typed, versioned toolpacks.
  • Make quality measurable, then improve it: Build the evals, benchmarks, and metric sets that score a run.

Ditto is redefining how data moves at the edge by providing a peer-to-peer sync engine for developers to build resilient, real-time applications without internet. With over $145 million in funding and trusted by organizations like Chick-fil-A and Delta Airlines, we are a globally distributed startup committed to building a diverse and inclusive team.

US Unlimited PTO

  • Architect and deploy autonomous AI agents and multi-agent workflows for privacy-first systems.
  • Build scalable backend services using FastAPI and orchestrate agentic workflows with LangGraph in AWS or Azure.
  • Develop rigorous evaluation pipelines for accuracy, citation adherence, latency, and reliability.

Osano is a leading data privacy platform that helps organizations comply with global privacy regulations like GDPR and CCPA. Backed by top-tier investors and recognized as a Great Place to Work for four years running, the company has a 97% employee satisfaction rate and a mission-driven, fast-growing culture.

$165,000–$210,000/yr
US

  • Lead the design and delivery of AI agents and automation workflows for NBCU's data collaboration ecosystem.
  • Guide development of reusable tools, APIs, and components for agentic and software engineering workflows.
  • Oversee production-grade Python application design, testing, and CI/CD practices to ensure reliability and scalability.

NBCUniversal is one of the world's leading media and entertainment companies, creating and distributing content across film, TV, streaming, and theme parks. As a subsidiary of Comcast, it fosters an inclusive culture and supports community engagement through employee-driven initiatives.

US Unlimited PTO

  • Collaborate with product and engineering teams to translate product objectives into autonomous agent-based solutions.
  • Design and build new agent data types, pipelines, and frameworks to coordinate reasoning, function calling, and actions.
  • Develop and optimize autonomous agents leveraging LLMs, planning algorithms, and multi-step reasoning approaches.

PointClickCare is a leading health tech company that helps providers deliver exceptional care. As a founder-led, privately held company with over 30,000 provider organizations and 400+ integrated partners, they are recognized by Forbes as a top private cloud company and honored as one of Canada's Most Admired Corporate Cultures, offering flexibility and growth opportunities.

US

  • Build and maintain production AI agents that power customer success workflows.
  • Define success metrics and evaluation frameworks before writing code.
  • Deploy, monitor, and iterate on agentic solutions to improve customer outcomes.

Samsara provides a Connected Operations Cloud platform that enables organizations to harness IoT data to improve safety, efficiency, and sustainability. As a recently public company, they foster a culture of autonomy and support in a hyper-growth environment focused on long-term impact.

India

  • Build and evolve production AI agents on foundation models using AWS Bedrock.
  • Develop evaluation pipelines and production observability for LLM systems.
  • Apply context engineering and agent architecture fundamentals to inform orchestration framework decisions.

Smart Working connects exceptional professionals with outstanding global teams through long-term remote opportunities. They are a high-rated workplace on Glassdoor that values integrity, excellence, and professional growth.

$80,000–$150,000/yr
US

  • Own a client's voice or chat agent end-to-end, from requirements through production and continuous iteration.
  • Engineer the agent's behavior by writing and maintaining instruction sets, and build automated evaluation gates.
  • Debug live conversations, work directly with US enterprise clients, and ship changes that impact real callers.

Rifa AI builds an AI agents platform for contact centers in regulated industries. They are a small team of passionate engineers with paying enterprise clients and growing revenue, backed by Seaborne Capital.

$100,000–$150,000/yr
US

  • You will design the architecture for specialized subagents operating within live customer conversations, including technical QA, product expertise, and objection handling.
  • You will build routing and delegation systems that determine when to answer directly, invoke a subagent, or escalate to a human.
  • You will master the dialogue platform, train AI agents via prompting and fine-tuning, and document workflows to educate the team.

1mind builds autonomous customer experience software that deploys AI-powered 'Superhumans' to engage, demo, onboard, and support customers across the entire buying journey. The company offers a remote-first, fast-moving culture with ownership, autonomy, and impact from day one.

$130,000–$175,000/yr
Global

  • Build and ship LLM-powered applications and AI agent workflows.
  • Design systems for reasoning, planning, memory, tool use and multi-step execution.
  • Build evaluation frameworks to measure AI quality and reliability.

Gina's Tech Jobs is an IT staffing and recruiting agency that specializes in placing technology professionals. They partner with companies to fill roles such as this one, offering a wide range of positions.

Europe

  • Own ML projects end-to-end from exploration to production deployment.
  • Build agentic LLM systems with multi-step workflows and tool use.
  • Treat evaluation as core engineering work with robust testing pipelines.

We are building AI to heal billions and unlock humanity's full potential, pioneering a new approach to healthcare. With over 1 million members, 15 million AI sessions, and $500 million raised, we are defining a new standard for healthcare.

Latin America

  • Design AI agent systems that collaborate with developers in real time.
  • Build and implement agentic systems for planning, coding, testing, and deployment.
  • Integrate LLMs and design orchestration layers for scalable AI systems on AWS.

Ubiminds connects Latin American tech professionals with software companies in the US and Canada. The company is GPTW-certified and has been supporting hundreds of professionals for 9 years.

Global 6w PTO 26w maternity 26w paternity

  • Own the roadmap for North's agent harness, including agent loop, context engineering, and tool orchestration.
  • Serve as the primary interface between North engineering and Cohere's Modeling team to ensure harness capabilities are validated before building.
  • Engage enterprise customers to surface real-world agentic failures and translate findings into product and model requirements.

Cohere is a security-first enterprise AI company that builds cutting-edge foundation models and end-to-end products for real-world business problems. The team consists of researchers, engineers, designers, and more, all passionate about their craft and dedicated to the widespread adoption of AI.

Global

  • Set and evolve the research direction for A1’s core intelligence, including context representation, memory, reasoning, planning, and orchestration.
  • Define evaluation frameworks that measure real-world usefulness, robustness, safety, and long-term behavior.
  • Own alignment, safety, and guardrail strategy as first-class product concerns.

A1 builds a proactive smart assistant for everyday users to bring intelligence to conversations, errands, organizing, and workflows. We are a small, high-talent-density team focused on shipping high-quality work and learning at rapid speed.

US Unlimited PTO

  • Evaluate AI systems at a scale only possible by combining thousands of vetted experts with model graders.
  • Innovate at the frontier of QA by shaping industry standards for validating agentic AI and large language models.
  • Collaborate with global market leaders to architect AI quality blueprints and drive high-impact consultative visibility.

Testlio provides a fully managed crowdsourced testing platform powered by proprietary intelligence technology, LeoCore. They are a female-founded, fully remote company with an inclusive culture, half of their team identifying as women, and are growing profitably.

India

  • Design and build the agent orchestration layer for agentic AI systems that manage workflow state and next-best-action logic.
  • Integrate LLMs behind a provider-agnostic layer with structured outputs, tool-calling, and model routing.
  • Collaborate with frontend, product, and client stakeholders to ship end-to-end solutions.

Monks is a global, digital-first, data-driven marketing and technology services company that combines marketing and technology services to redefine brand interactions. With a culture of innovation, it ranks among Cannes Lions' Top 10 Creative Companies and fosters diversity and inclusion.

Global

  • Own and build evaluation pipelines for AI workflows to define metrics, benchmarks, and acceptance criteria.
  • Drive continuous improvement of AI performance through quantitative analysis, prompting strategies, and workflow design.
  • Ensure reliability, monitoring, and stability of AI systems with dashboards, alerts, and failure detection.

HiPeople is an AI Hiring Platform that automates screening, interviews, assessments, and references for recruiting teams. They are a lean startup backed by leading investors, aiming to reach $10M ARR with fewer than 20 people, and work with brands like the NFL and Zapier.

Latin America

  • Design and deploy autonomous agentic AI workflows and multi-agent systems for production.
  • Architect and optimize advanced RAG pipelines using hybrid search and semantic routing.
  • Implement enterprise safety controls and HIPAA-compliant AI pipelines.

AspenView Technology Partners specializes in creating high-performing nearshore IT teams to help North American clients innovate faster and more efficiently. They are an equal opportunity employer that values diversity and inclusion.

US

  • Own AI systems end to end: from ingestion to serving, including retrieval and agentic workflows.
  • Build production RAG and semantic search systems grounded in the warehouse.
  • Design and ship self-service AI tooling and agentic workflows for 200+ internal stakeholders.

We are a healthcare technology company connecting people and partners with coverage, care, and solutions. We are curious, collaborative, and always looking for better ways to solve problems.