Source Job

Canada

  • Lead AI validation strategies for LLMs, RAG systems, and AI agent workflows.
  • Build automated validation pipelines and evaluation frameworks for AI output quality.
  • Establish production monitoring, AI guardrails, and quality metrics for AI products.

Node.js MongoDB LLM Evaluation Prompt Engineering CI/CD

20 jobs similar to Senior Quality Engineer (AI)

Jobs ranked by similarity.

Europe Unlimited PTO

  • Own and drive the quality, reliability, and evolution of AI systems in production across multiple products.
  • Design and orchestrate agentic AI workflows, tool-based systems, and multi-step LLM chains with clean tool contracts.
  • Manage production prompt systems, run A/B experiments, and monitor AI system metrics using Langfuse and other observability tools.

Ruby Labs is a leading tech company that creates and operates innovative consumer products across health, education, and entertainment. The company has a fast-growing, ambitious team with a high-performance culture.

Global

  • Own and build evaluation pipelines for AI workflows to define metrics, benchmarks, and acceptance criteria.
  • Drive continuous improvement of AI performance through quantitative analysis, prompting strategies, and workflow design.
  • Ensure reliability, monitoring, and stability of AI systems with dashboards, alerts, and failure detection.

HiPeople is an AI Hiring Platform that automates screening, interviews, assessments, and references for recruiting teams. They are a lean startup backed by leading investors, aiming to reach $10M ARR with fewer than 20 people, and work with brands like the NFL and Zapier.

Canada

  • Establish quality engineering practices for AI-powered chatbot products, including testing strategies and release processes.
  • Design evaluation methods for non-deterministic LLM systems, focusing on reliability, performance, and safety.
  • Collaborate with engineers and product teams to integrate quality throughout development and mentor peers.

The company specializes in AI-powered conversational products, building next-generation chatbot capabilities. They work in multidisciplinary teams with a focus on reliability and innovation in a remote-first environment.

LATAM

  • Build and ship MVPs fast, working hands-on with LLMs, APIs, and AI-assisted workflows.
  • Use tools like Copilot, Cursor, GPT, and Claude daily to ship AI-powered product inside high-growth startups.
  • Work fully remote with flexible schedules, part-time or side gig, with some US time overlap.

Futureproofing is a talent platform focused on embedding high-caliber engineers into startups building AI-driven products. They match intentionally, not through keyword filtering, and focus on output, ownership, and shipping speed in a developer-first model.

Brazil

  • Lead complex AI engineering workstreams across multiple business areas and stakeholder groups.
  • Design and build AI-powered applications using LLMs, agents, and modern architectures.
  • Establish best practices for prompt engineering, API integrations, and code quality.

The partner company focuses on designing and delivering advanced AI solutions that help organizations transform business processes and create measurable value. It offers a collaborative and inclusive culture with opportunities for career growth and international mobility.

US

  • Design and build advanced AI-powered systems including autonomous agents and generative AI capabilities.
  • Lead architecture and implementation of intelligent systems that improve digital learning experiences.
  • Collaborate with cross-functional teams to bring AI solutions from concept to production.

The company builds AI-powered digital learning platforms that transform educational experiences. They serve millions of learners and educators with a focus on responsible AI and inclusive culture.

US

  • Evaluate AI quality across the advisor stack, including pre-call briefs, in-call guidance, and post-call outputs.
  • Iterate inside ORA by refining prompts, updating knowledge base entries, and tweaking skills to close the loop on issues.
  • Surface trends and drive continuous improvement by tagging conversations, logging issues, and recommending prioritized improvements.

HighLevel is an AI-powered business operating system that gives agencies, entrepreneurs and SMBs the infrastructure to build, automate and scale. With over 2,000 team members across 10+ countries, HighLevel operates as a global, remote-first organization built for speed and ownership.

US Unlimited PTO

  • Design and build agentic AI systems using frameworks like Mastra and n8n to power generative video platforms.
  • Work hands-on with creative and product teams to translate ideas into working AI architectures in real time.
  • Own observability, evaluation, and prompt management to ensure high-quality production performance.

Misfits & Machines is the innovation sister company of Marketing Architects, building transformative AI technologies to help marketers achieve remarkable growth. They are a fully remote team with industry-leading benefits and a culture focused on bold ideas and precision execution.

India

  • Design and develop AI-driven QA systems that autonomously explore applications and workflows.
  • Build intelligent testing solutions that generate and execute tests dynamically based on product behavior.
  • Establish continuous testing loops to identify and resolve issues throughout the development lifecycle.

The company focuses on building AI-native and agentic systems for next-generation software quality engineering. It offers a remote-first work environment with a culture centered on innovation, learning, and continuous improvement.

India

  • Lead the development of test automation frameworks across UI, API, and GraphQL layers, focusing on reliability and speed.
  • Design and integrate AI evaluation frameworks to assess accuracy and reliability of LLM-powered features.
  • Optimize test execution pipelines for faster feedback in CI/CD environments and mentor QE engineers on best practices.

AlphaSense provides AI-driven market intelligence and search to help enterprises make smarter decisions. Founded in 2011 and headquartered in New York City, the company has over 2,000 employees globally and serves more than 6,000 customers, including a majority of the S&P 500.

$225,000–$245,000/yr
US Unlimited PTO

  • Lead the quality assurance function across the full software development lifecycle, including strategy, organizational design, and technical standards.
  • Drive an AI-driven technical quality architecture with test data strategy, CI/CD integration, and automation framework philosophy.
  • Partner with platform and MLE teams to integrate AI/LLM evaluation into the delivery pipeline and establish release quality standards.

Machinify is a healthcare intelligence company that provides an AI-powered platform to health plans, maximizing financial outcomes and reducing costs. Deployed by over 85 health plans and representing over 270 million lives, the company fosters a culture of innovation and transparency.

$119,981–$179,971/yr
US Unlimited PTO

  • Develop and integrate AI-powered systems to support EdReports' curriculum review process, including data pipelines, retrieval systems, and prompt workflows.
  • Build internal tools and interfaces for users to interact with AI outputs, ensuring accessibility and security.
  • Collaborate with product managers and subject matter experts to translate user needs into scalable technical solutions.

EdReports.org is a nonprofit that publishes free, online, evidence-rich reviews of instructional materials to improve curriculum quality. They have a remote-first culture and are committed to diversity, equity, and inclusion.

Global

  • Ship the agent platform to production: event-triggered queue, model-agnostic runtime, durable state, human review gate, atomic rollback, and complete run logging.
  • Own the evaluation layer with golden suites, behavioral assertions, safety cases, and a CI gate that blocks regressions before merging.
  • Enforce governance in code: risk tiers, least-privilege credentials, permission gates, audit log, and autonomy classes where dangerous actions are structurally impossible.

Supabase is the Postgres development platform providing a complete backend solution including Database, Auth, Storage, Edge Functions, Realtime, and Vector Search. With ~400 team members across 60+ countries, we are a born-remote, open-source-first company that builds tools developers love.

$104,244–$134,904/yr
Canada

  • Define product direction and build AI voice agent experiences for drivers.
  • Own technical architecture for coordinating multiple AI agents and real-time interactions.
  • Mentor engineers and champion Samsara's cultural principles while shipping high-impact features.

Samsara is the pioneer of the Connected Operations Cloud, using IoT data to improve safety, efficiency, and sustainability for physical operations. As a recently public company, they have a global team and a culture focused on customer success, long-term thinking, and inclusivity.

Global

  • Design and build the AI execution platform with event-triggered workflows and model-agnostic runtimes.
  • Develop evaluation layers with golden test suites and safety checks to ensure AI reliability.
  • Implement governance mechanisms and build AI agents supporting business workflows.

Jobgether is a platform that connects talent with opportunities using AI-powered matching. The company has a globally distributed team and a remote-first culture.

$150,000–$165,000/yr
US Unlimited PTO

  • Establish quality engineering strategy for AI-driven systems, integrating validations into CI and release pipelines.
  • Collaborate early with engineering to surface risks and promote testability as a core design requirement.
  • Expand verification approach beyond unit tests into integration, system, stress, and full workflow testing.

Dropzone AI scales cybersecurity beyond human limits by augmenting security engineers with AI specialists. It is an award-winning, venture-backed startup disrupting the $200B+ cybersecurity market with a team experienced in cybersecurity, AI/ML, and SaaS.

Canada Unlimited PTO

  • Lead quality for AI-powered experiences by designing and maintaining automated test strategies for Voice AI agents, LLM-powered workflows, and cloud-native applications.
  • Build automation that scales by developing frameworks, integrating testing into CI/CD pipelines, and improving quality processes.
  • Validate complex distributed systems through end-to-end, API, UI, integration, and performance testing across microservices and cloud platforms.

Phreesia is a SaaS platform that digitizes patient intake, engagement, and care experiences for healthcare organizations. A five-time Modern Healthcare Best Places to Work winner and recognized on the Bloomberg Gender Equality Index, they employ a diverse and inclusive workforce with a focus on innovation and employee growth.

US

  • Lead a high-performing AI and data science team to develop and deploy AI-driven security systems.
  • Define and execute the technical roadmap for AI, ML, and data systems, ensuring robust performance and scalability.
  • Manage the entire lifecycle of data pipelines and AI models, integrating with platforms like AWS Bedrock and OpenAI.

Bugcrowd is a crowdsourced security platform that unites organizations with a global network of hackers to identify vulnerabilities. The company is backed by prominent investors and fosters a collaborative, inclusive culture with a diverse team of professionals.

$48,000–$60,000/yr
US

  • Own testing for document products and AI-assisted tools, ensuring correctness, reliability, and edge case validation.
  • Write and maintain end-to-end tests, balancing manual and automated testing approaches.
  • Collaborate with cross-functional teams and partner with the Forward Deployed team to QA client portals.

Pasito is the AI workspace for employee benefits, rethinking group insurance and benefits underwriting, delivery, and management. Backed by Y Combinator and Core Innovation Capital, they are a fast-growing, collaborative team building accurate and scalable systems.

Brazil

  • Define and advance quality engineering practices across complex software environments.
  • Build and evolve automation frameworks, incorporating AI technologies like LLMs and autonomous agents.
  • Collaborate with multidisciplinary teams and enterprise clients to ensure high-quality, reliable systems.

This partner company creates impactful digital products used by millions of people across multiple platforms and devices. It values innovation, continuous learning, and proactive problem-solving, offering a collaborative and growth-oriented environment.