Source Job

Mexico Brazil Argentina

  • Architect and scale automated evaluation pipelines for multi-agent LLM systems.
  • Define domain-specific metrics using advanced evaluation techniques like LLM-as-a-judge.
  • Build benchmarking infrastructure and monitor live agent workflows for drift and cost.

Python LLM Test Automation LangChain AWS

20 jobs similar to Senior Evaluation AI Engineer

Jobs ranked by similarity.

Global

  • Design and build automated evaluation pipelines for AI agents.
  • Develop testing frameworks for reasoning, multi-turn conversations, and adversarial scenarios.
  • Own the evaluation architecture and integrate observability tools in a fully remote team.

A company building production-grade AI agents and systems. They operate in a fully remote, globally distributed environment with a culture focused on innovation, autonomy, and continuous improvement.

Europe 6w PTO

  • Own agent quality end-to-end for Bliro's chat and voice agents, including prompting, context, and model evaluation.
  • Build eval sets, LLM judges, and benchmarks to measure quality, latency, and cost with statistical rigor.
  • Establish a repeatable process for improving agent quality and set the long-term technology roadmap.

Bliro builds an AI assistant that handles desk work for field sales teams, with over one million customer touchpoints documented. Backed by leading investors and trusted by German Mittelstand companies, it's a small, high-performing team that values ownership and continuous improvement.

AI Engineer

Jeeves
Latin America

  • Own AI workflows end to end, from problem definition to production and improvement, for finance processes like invoice-to-payment and reconciliation.
  • Design and build production-grade LLM-based systems with structured outputs, RAG, validation, and monitoring.
  • Collaborate with product, backend engineering, and customers while setting technical direction and using AI coding tools daily.

Jeeves is a stablecoin-native banking platform providing corporate cards, payments, treasury, and spend management for global enterprises. With over $5B in annualized volume and backing from top investors, the company is scaling rapidly and values relentlessness and tackling hard problems.

$136,908–$185,000/yr
North America

  • Lead the AI Core team as Staff AI Engineer, setting technical direction for Compass, the AI agent platform.
  • Architect multi-agent systems, own MCP/A2A integrations, and drive agent orchestration, evaluation, and observability.
  • Oversee large-scale technical projects, mentor engineers, and champion AI-assisted development across the organization.

Basis empowers agencies and brands with software that automates digital media operations, including planning, reporting, and financial reconciliation. Headquartered in Chicago, it is a top workplace with a culture focused on inclusivity and growth.

  • Building multi-agent systems and implementing agent-based workflows and AI automation solutions.
  • Developing and optimizing prompt engineering strategies, including evaluation and iteration.
  • Integrating LLMs and AI models into cloud-based applications and APIs while researching new AI tools.

Software Mind develops impactful software solutions for global companies, partnering with tech giants and unicorns on transformative projects. The company builds cross-functional engineering teams and embraces a culture of openness, respect, grit, and cooperation.

Brazil

  • Design, develop, and maintain AI agents and LLM-powered applications using Python and AWS services.
  • Implement prompt engineering, agent orchestration, and evaluation frameworks to ensure reliable, scalable AI outputs.
  • Collaborate with engineering and product teams to deliver production-ready AI solutions with robust guardrails and observability.

CI&T helps large enterprises transform the potential of AI into real business impact with AI deployment, AI-native execution, and tech-integrated business solutions. With 30 years of experience and 8,000 employees across more than 25 countries, they foster a collaborative culture where AI is embedded in how they work and innovate every day.

$104,074–$123,889/yr
Latin America 4w PTO 10w maternity

  • Build and evolve Superhuman Mail's MCP layer with new tools and standards.
  • Design and ship agentic workflows and personalization features at scale.
  • Build evaluation systems that measure quality and catch regressions as AI changes.

Superhuman is an AI productivity platform integrating with over 1 million apps to eliminate busywork. Founded in 2009, it empowers 40 million people and 50,000 organizations with a tight-knit, collaborative culture.

US

  • Own quality for non-deterministic, multi-agent AI systems, building the evaluation harness that underpins AI governance.
  • Verify grounding and faithfulness, building automated checks to stop hallucinations from reaching production.
  • Red-team for prompt injection, jailbreaks, and tool misuse, and prove multi-tenant data isolation.

SOFICO Intelligence uses AI to extract value from back-office systems, building agentic AI for financial, leasing, fleet, and mobility data. Backed by global ERP leader SOFICO, it's a remote-first team of curious, self-directed builders.

Brazil

  • Design and build AI agents and LLM-powered applications using Python and AWS Bedrock.
  • Develop evaluation frameworks to ensure agent accuracy, reliability, and performance.
  • Optimize LLM workflows, observability, and operational costs for production-ready AI solutions.

The company is building an AI-powered media planning platform that leverages large language models and AWS technologies. The remote-first team focuses on innovation, collaboration, and delivering scalable AI solutions.

Global

  • Design and ship features end to end across data model, API, background workers, and UI.
  • Build and improve agent and model evaluation: sandboxed trials, scoring, and trustworthy aggregation of thousands of runs.
  • Improve benchmark task quality checks, integrate new models and agents, and keep the system fast and reliable as usage grows.

Turing accelerates superintelligence by providing high-quality datasets, reinforcement learning environments, and research benchmarks to frontier AI labs, and also deploys agentic AI systems for Fortune 500 enterprises. Headquartered in San Francisco, it operates with startup speed and an AI-forward culture.

$180,000–$225,000/yr
US

  • Design and build LLM-powered agents and workflows including orchestration, tool use, retrieval, structured extraction, and human-in-the-loop review.
  • Build evaluations with data labelers and subject matter experts, running them on every change to ensure reliability.
  • Make agents production-grade by implementing guardrails, observability, cost and latency optimization, and careful handling of sensitive health data.

CCC Intelligent Solutions Inc. is a leading cloud platform for the multi-trillion-dollar insurance economy, creating intelligent experiences for insurers, repairers, automakers, and more. With over 35,000 businesses using its technology, the company focuses on innovation and collaboration to keep lives moving forward.

Latin America

  • Ship end-to-end AI products: transform ambiguous business challenges into scalable, production-grade AI features and applications.
  • Build persistent and compliant AI memory architectures using vector stores like Pinecone and Pgvector.
  • Unify fragmented enterprise data by developing pipelines integrating CRM, ERP, and marketing platforms for reliable AI agent sources.

Nortal is a global digital transformation company that delivers complex solutions for enterprises and public sector organizations. With over 25 years of experience, it fosters a culture of autonomy, open communication, and respect for diversity, working as a close-knit remote team across LATAM.

US

  • Research and define end-to-end AI system requirements from capability to behavior to user impact.
  • Work closely with ML, backend, and mobile engineers on system design, evaluation, and iteration.
  • Define and evolve evaluation frameworks across offline metrics, online experiments, and human feedback.

AI Engineer

Elicit
$185,000–$320,000/yr
US 4w PTO

  • Build evaluations to measure and improve AI reasoning in complex, high-stakes decisions.
  • Collaborate with engineers, product, and domain experts to turn observed failures into testable improvements.
  • Own the quality of shipped systems, from code reviews to monitoring and integration.

Elicit builds AI systems that help research and business teams make evidence-based decisions on high-stakes questions. They are a remote-friendly, collaborative team that values rigorous thinking, experimentation, and in-person retreats.

Global

  • Own quality for Nexxa's AI agents: build eval harnesses, simulators, and CI/CD quality gates.
  • Perform manual and exploratory testing, trace reviews, red-teaming, and release verification.
  • Define metrics and datasets, mentor engineers, and champion testability for LLM-driven products.

Nexxa.AI builds AI systems for heavy industry, enabling autonomous operations across manufacturing, logistics, and infrastructure. The team values innovation, discipline, and continuous improvement, with strong growth opportunities.

$170,000–$220,000/yr
US

  • Own the context layer: manage what Mimo knows, when, and at what cost, with retrieval and memory as one budget.
  • Define tool contracts and build evaluation practice: schemas, results, failure modes, and regression gates for every change.
  • Shape how Mimo reasons, asks, explains, and recovers, and decide which model runs where.

MimoLabs builds Mimo, an agentic system for making physical things, from design to production across 3D printing, CNC, and laser cutting. We are a small, VC-funded, experienced team based in Detroit and remote across the US, fast, adaptable, and honest.

Global 7w PTO

  • Build and own evaluation infrastructure for Circle's AI agents.
  • Diagnose quality breakdowns across the agent pipeline.
  • Grow datasets and lead the AI Quality engineering team.

Circle builds an AI-powered all-in-one platform for digital businesses, uniting audiences with discussions, live streams, events, chat, courses, and payments. It's a fully remote team of about 270 people from 30+ countries, with an async culture and annual global offsites.

US Unlimited PTO 16w maternity 16w paternity

  • Build shared SDKs, APIs, and infrastructure for AI features and agentic workflows.
  • Integrate durable execution infrastructure for long-running agent workflows, including checkpointing, retries, and human-in-the-loop steps.
  • Develop evaluation and debugging tools that turn production failures into measurable improvements and regression coverage.

Vanta helps businesses automate security monitoring and demonstrate trust through compliance with standards like SOC 2, HIPAA, and ISO 27001. Founded in 2018, it has a kind, talented team and thousands of customers worldwide.

$33–$44/hr
EU/EEA

  • Design, build, and deploy AI agents that automate real business processes.
  • Integrate LLMs from OpenAI, Gemini, Anthropic, and open-source models into products.
  • Own delivery end-to-end, from prototype to production, choosing the right LLM/tooling.

We are a software development agency based in Vilnius, Lithuania, building production-ready AI and software solutions. Our team works remotely across the EU/EEA, with a focus on independent, fast-moving delivery.

Latin America

  • Combine hands-on AI engineering with platform support to ensure teams can effectively use approved AI tools and internal solutions.
  • Monitor platform health, usage, and adoption, ensuring AI tools remain reliable and effective for business teams.
  • Design, build, and maintain custom integrations, Slackbots, and AI agents that connect enterprise AI tools with internal content, systems, and workflows.

The partner company distributes AI platforms across international organizations, focusing on security and IT. It operates with a small distributed LATAM team and supports US-based clients, offering a collaborative remote environment.