Design, implement, and operate benchmark execution and evaluation harnesses for AI models and agentic workflows.
Develop evaluation methodologies that combine automated metrics with structured human subject matter expert judgment.
Produce defensible evaluation reports comparing candidate capabilities with current mission workflows, including documented limitations and failure modes.
Evaluate AI systems at a scale only possible by combining thousands of vetted experts with model graders.
Innovate at the frontier of QA by shaping industry standards for validating agentic AI and large language models.
Collaborate with global market leaders to architect AI quality blueprints and drive high-impact consultative visibility.
Testlio provides a fully managed crowdsourced testing platform powered by proprietary intelligence technology, LeoCore. They are a female-founded, fully remote company with an inclusive culture, half of their team identifying as women, and are growing profitably.
Develop and deploy machine learning and AI systems.
Work with LLMs, generative AI, and modern ML frameworks.
Optimize model performance, latency, and cost.
A fast-growing technology company building critical infrastructure that powers high-volume, real-time business operations across multiple systems and platforms. It is a collaborative, fast-moving environment where engineers have meaningful influence on architecture and product direction.
Build and maintain evaluation pipelines and environments for AI model benchmarks.
Prepare and maintain benchmark datasets, ensuring reproducibility and consistency.
Develop analysis tools and leaderboards to track model performance over time.
Office Hours is an on-demand expert network that connects leading organizations with trusted experts across various knowledge domains. We're a hyper-growth and profitable company, quickly expanding our expert network, launching new offices, and new products.
Participate in a structured training track to develop expertise in AI benchmarking, profiling, and performance tuning.
Build and scale benchmarking infrastructure for evaluating AI systems in enterprise settings.
Design agent evaluation pipelines that measure reasoning, accuracy, alignment, and user outcomes.
DevRev is building Computer, an AI teammate that unifies data, tools, and workflows into a single AI-ready platform, giving employees real-time insights and proactive suggestions. Backed by Khosla Ventures and Mayfield with over $150M raised, the company is trusted by global companies across industries and fosters a culture of innovation and collaboration.
Architect and deploy autonomous AI agents and multi-agent workflows for privacy-first systems.
Build scalable backend services using FastAPI and orchestrate agentic workflows with LangGraph in AWS or Azure.
Develop rigorous evaluation pipelines for accuracy, citation adherence, latency, and reliability.
Osano is a leading data privacy platform that helps organizations comply with global privacy regulations like GDPR and CCPA. Backed by top-tier investors and recognized as a Great Place to Work for four years running, the company has a 97% employee satisfaction rate and a mission-driven, fast-growing culture.
Build and ship LLM-powered applications and AI agent workflows.
Design systems for reasoning, planning, memory, tool use and multi-step execution.
Build evaluation frameworks to measure AI quality and reliability.
Gina's Tech Jobs is an IT staffing and recruiting agency that specializes in placing technology professionals. They partner with companies to fill roles such as this one, offering a wide range of positions.
Design and run experiments to understand model behavior and test new ideas.
Create datasets, benchmarks, and evaluation methods for hard problems.
Work directly with AI labs to turn open-ended goals into concrete research projects.
Vetto builds the infrastructure for next-generation AI training data. They partner with the world’s top AI labs and offer a fast, flat, remote-first culture with high ownership.
Own the AI foundation product teams build on: model selection, routing, context management, and tool design.
Own eval infrastructure by defining output quality standards with customers and encoding them in evals.
Ship agentic features end to end from v0 through optimization, including prompt engineering, caching, and parallel tool calls.
Aleph is an AI-native platform for Financial Planning & Analysis (FP&A), solving data scattered across systems and spreadsheets. Backed by top VCs, it works with customers like Webflow and Notion, and is founded by former Google and Bain engineers.
Build the agent execution runtime: Design and ship the layer that compiles a brief into a running team of agents.
Design the tool surface agents work through: Replace "here is a shell, please be careful" with typed, versioned toolpacks.
Make quality measurable, then improve it: Build the evals, benchmarks, and metric sets that score a run.
Ditto is redefining how data moves at the edge by providing a peer-to-peer sync engine for developers to build resilient, real-time applications without internet. With over $145 million in funding and trusted by organizations like Chick-fil-A and Delta Airlines, we are a globally distributed startup committed to building a diverse and inclusive team.
Own the development, fine-tuning, and evaluation of in-house AI models from dataset design through to production deployment.
Run supervised fine-tuning and post-training experiments, establishing benchmarks and evaluation harnesses.
Take models to production on Stream's serving stack, tuning for latency, cost, and reliability at high volume.
Stream provides APIs and SDKs for building activity feeds, chat, and voice/video, powering over a billion end users worldwide. We are a team of over 120 peers from over 35 countries, with a casual social culture that values transparency and excellence.
Drive AI platform architecture and execute the long-term roadmap for machine learning and generative AI.
Lead AI infrastructure vision including end-to-end training, fine-tuning, and low-latency inference platforms.
Set engineering excellence standards for the full AI/ML SDLC and mentor senior engineers.
Lyra Health is a leading provider of evidence-based mental health care, serving over 20 million people globally. The company operates with a focus on transformative care and has delivered over 15 million sessions, employing a large team that includes engineers, data scientists, and clinical professionals.
Own ML projects end-to-end from exploration to production deployment.
Build agentic LLM systems with multi-step workflows and tool use.
Treat evaluation as core engineering work with robust testing pipelines.
We are building AI to heal billions and unlock humanity's full potential, pioneering a new approach to healthcare. With over 1 million members, 15 million AI sessions, and $500 million raised, we are defining a new standard for healthcare.
Evaluate AI coding-agent interactions for technical accuracy and engineering judgment.
Assess explanations and reasoning to ensure they genuinely help developers.
Provide structured feedback to improve AI-assisted development experience.
Jobgether is an AI-powered job matching platform that connects candidates with hiring companies through objective AI-based review. The role is posted on behalf of a partner company that develops AI coding tools.
Lead rapid prototyping and technical validation of agentic AI solutions and autonomous workflows.
Evaluate foundation models, agent frameworks, and orchestration technologies for enterprise adoption.
Mentor AI engineers and promote strong engineering practices across teams.
This partner company is hiring for a senior technical leadership role focused on shaping next-generation AI agents and autonomous workflows. They are a collaborative enterprise with a culture centered on experimentation, continuous learning, and innovation, offering a remote opportunity in the US.
Lead the design and delivery of AI agents and automation workflows for NBCU's data collaboration ecosystem.
Guide development of reusable tools, APIs, and components for agentic and software engineering workflows.
Oversee production-grade Python application design, testing, and CI/CD practices to ensure reliability and scalability.
NBCUniversal is one of the world's leading media and entertainment companies, creating and distributing content across film, TV, streaming, and theme parks. As a subsidiary of Comcast, it fosters an inclusive culture and supports community engagement through employee-driven initiatives.
Design and iterate on agent behavior across real, live workflows, including long-horizon, multi-turn agentic tasks.
Build and run evaluations against production conditions to measure performance, regressions, and edge cases.
Partner with Product and Engineering to ensure agents are steerable, trustworthy, and ready for scale.
Arcadia is a healthcare platform that transforms complex data into trusted intelligence for providers, payers, and life sciences organizations. Backed by Nordic Capital, the company has hundreds of organizational clients and a mission-driven culture focused on improving patient outcomes.
Own and evolve an AI evaluation harness to rigorously test AI agents and non-deterministic outputs.
Design automated evaluations, adversarial test cases, and benchmark comparisons to expose edge cases and logic failures.
Validate regulatory quality gates and compliance requirements, including frameworks such as SR 26-2 and NYDFS Regulation 187.
Jobgether is an AI-powered job matching platform that connects candidates with hiring companies. It uses AI to review applications and share top-fitting candidates with employers, focusing on efficient and fair recruitment.
Own the shared AI foundation used by product engineering teams, including model selection, routing, context management, and evaluation.
Design and ship agentic capabilities end to end, from proof-of-concept through production optimization and iterative improvement.
Establish evaluation infrastructure and practices to measure model and agent performance, ensuring quality and accuracy for financial decision-making.
Our partner is building a suite of financial planning and analysis capabilities powered by a shared AI foundation. They operate as a remote-first engineering team with a focus on autonomy, craftsmanship, and high-impact work.
Own the ML/AI platform, including training infrastructure, model serving, inference pipelines, and production integrations.
Set technical direction for ML/AI architecture, tooling, standards, and strategy in partnership with data science and engineering.
Drive GenAI and LLM engineering, including RAG, prompt engineering, fine-tuning, and agentic workflows.
This company builds AI/ML infrastructure for financial technology products used by over 1,600 financial institutions. It operates as a globally distributed team with a focus on innovation and collaboration.
Provide technical leadership for designing scalable, AI-integrated software solutions and guiding a distributed team.
Architect robust AI agents and APIs, refine prompt strategies, and decompose complex features into clear tasks.
Mentor developers on AI tool adoption and clean code practices, optimizing DevOps pipelines for efficiency.
Mitratech builds world-class products simplifying operations in Legal, Risk, Compliance, and HR functions. They serve 20,000 client companies globally, including 30% of the Fortune 500, with a diverse and inclusive culture focused on learning and fun.