Source Job

$160,000–$210,000/yr
US

  • Build and maintain evaluation pipelines and environments for AI model benchmarks.
  • Prepare and maintain benchmark datasets, ensuring reproducibility and consistency.
  • Develop analysis tools and leaderboards to track model performance over time.

Python Docker React Next.js Machine Learning

20 jobs similar to Software Engineer, Benchmarking

Jobs ranked by similarity.

US

  • Develop and evaluate AI training data for LLM and AI agent platforms.
  • Create coding tasks and write reference-quality solutions for evaluation.
  • Critically assess AI-generated code for correctness, security, and maintainability.

Toloka is a leading expert human data platform for AI agents and LLMs, providing high-quality training data. The company focuses on improving AI models through human feedback and structured evaluation.

US

  • Lead applied AI projects from concept to impact, building and deploying ML/GenAI solutions across product, engineering, and security teams.
  • Act as an internal AI consultant, scoping problems, evaluating approaches, and advising on best practices for generative technologies.
  • Design experiments, build models for anomaly detection and fraud analysis, and bridge research to production via scalable APIs and tools.

Sonatype is a software supply chain management company that invented componentized software development and pioneered the category. Trusted by over 2,000 organizations, including 70% of the Fortune 100, and 15 million developers, the company fosters innovation, AI/ML adoption, and an inclusive culture.

$138,500–$225,500/yr
US 16w maternity 16w paternity

  • Design, train, and ship ML systems for governance and security like anomaly detection and trust scoring.
  • Build data pipelines, model serving, evaluation frameworks, and feedback loops.
  • Set technical direction, own architecture, and help recruit and mentor as the team grows.

Docker provides developer tooling trusted by over 20 million monthly users and billions of container pulls. They are a globally distributed, remote-first team building tools for software delivery.

Global

  • Evaluate LLM architecture logic for technical accuracy and audit ML code and notebooks for efficiency.
  • Refine RLHF frameworks to align models with human intent and analyze model reasoning in complex chain-of-thought prompts.
  • Benchmark performance by conducting comparative testing between model outputs based on technical metrics.

Prolific connects researchers with a global pool of participants for collecting high-quality human data to train AI models. With over 35,000 users, they focus on ethical data gathering to advance AI capabilities.

US

  • Design, build, and own shared infrastructure for Interpretability research environments, data systems, and compute tooling.
  • Lead cross-team efforts with agentic engineering, security, compute, and storage platform teams.
  • Discover and resolve major organization-wide developer experience issues and help take interpretability methods from research code to dependable audit pipelines.

Anthropic is an AI safety company focused on building reliable, interpretable, and steerable AI systems. They are a quickly growing team of researchers, engineers, policy experts, and business leaders committed to beneficial AI.

Global

  • Design and build backend services for AI-powered product features, including inference pipelines and orchestration layers around LLMs.
  • Develop high-throughput, low-latency distributed systems with monitoring, logging, and alerting across production services.
  • Collaborate with product, infrastructure, and AI engineers to optimize performance, caching, batching, and streaming.

This team is building an AI-native productivity platform that replaces repetitive digital work with reliable AI workflows. They are a small, focused product team working on cutting-edge AI infrastructure.

$165,000–$330,000/yr
US Unlimited PTO

  • Partner directly with customers to architect, build, and deploy high-scale production AI applications on Baseten's platform.
  • Own the journey from initial exploration to production deployment, translating ambiguous goals into reliable services.
  • Work across product, software development, performance engineering, and customer-facing implementations.

Baseten powers mission-critical inference for dynamic AI companies like Cursor and Notion. They are rapidly growing, recently raised a $1.5B Series F, and foster a collaborative, forward-thinking culture.

$153,000–$180,000/yr
US

  • Lead end-to-end development of advanced AI solutions including agentic systems, RAG pipelines, and multi-agent workflows.
  • Architect AI infrastructure with guardrails, access controls, and compliance protocols for ethical deployment.
  • Guide multi-disciplinary teams, setting technical playbook and best practices.

Nava is a consultancy and public benefit corporation that makes government services simple and effective. Since 2015, federal, state, and local agencies have trusted Nava to solve technology modernization challenges, and they maintain a collaborative, remote-friendly team environment.

Global

  • Design and maintain LLM-powered backend services using Python and FastAPI.
  • Implement retrieval-augmented generation (RAG) for structured and unstructured fleet data.
  • Optimize retrieval accuracy, latency, and hallucination rates through automated evaluation pipelines.

Datakrew revolutionizes EV fleet intelligence with IoT and AI solutions. They aim to serve one million EVs within 5 years and cultivate a mission-driven culture.

Unlimited PTO

  • Build the environments and verifiers our models train against
  • Own the synthetic data pipeline, from generation through quality gates
  • Ship models into production and keep improving them on real usage

LiveKit builds the infrastructure layer for the voice-driven era of computing, enabling developers to build, deploy, and scale voice AI applications. Founded in 2021, they power voice AI for major companies and have a small, senior team that values craft and creativity.

$161,000–$221,500/yr
United States

  • Drive AI platform architecture and execute the long-term roadmap for machine learning and generative AI.
  • Lead AI infrastructure vision including end-to-end training, fine-tuning, and low-latency inference platforms.
  • Set engineering excellence standards for the full AI/ML SDLC and mentor senior engineers.

Lyra Health is a leading provider of evidence-based mental health care, serving over 20 million people globally. The company operates with a focus on transformative care and has delivered over 15 million sessions, employing a large team that includes engineers, data scientists, and clinical professionals.

$230,000–$270,000/yr
US

  • Design, build, and maintain core AI capabilities that power intelligent experiences across the platform.
  • Develop reusable capabilities for model integration, reasoning, and evaluation.
  • Optimize AI systems through experimentation, quantitative evaluation, and iterative refinement.

Legion embeds intelligence inside complex systems, unlocking data and accelerating human workflows for government and enterprise. They collaborate with world-class partners like Palantir and Nvidia, building intelligent infrastructure with a team of bold thinkers.

US

  • Design and build sandboxed evaluation environments for AI models to safely execute code and interact with tools.
  • Build backend services and infrastructure supporting large-scale AI and agentic evaluations.
  • Develop agent scaffolding, evaluation harnesses, and systems for provisioning isolated environments using Docker, Kubernetes, and cloud infrastructure.

10a Labs is the safety and threat-intelligence layer trusted by frontier AI labs, AI unicorns, Fortune 10 companies, and leading global technology platforms. The company focuses on adversarial red teaming and model evaluations to help teams deploy AI systems safely.

UK 6w PTO

  • Build AI-powered investigation tools including Slack bots and agents that analyze support tickets and query production logs.
  • Develop data infrastructure for ClickHouse analytics, real-time anomaly detection, and platform health monitoring.
  • Create full-stack internal tooling that bridges data, AI, and the people who need answers.

Reactive Markets is the 2026 OTC Trading Platform of the Year, handling over $50 billion in daily trading volumes across FX, Equities, and Cryptocurrency. Their small, senior engineering team builds cutting-edge, high-performance trading systems and values transparency, honest feedback, and documentation.

Global 6w PTO 26w maternity 26w paternity

  • Design and write high-performing scalable software for training models.
  • Develop new tools to support and accelerate research and LLM training.
  • Collaborate with engineering teams and scientific teams to implement experiments on cluster and data infrastructure.

Cohere is a security-first enterprise AI company building cutting-edge foundation AI models and end-to-end products for real-world business problems. The company is a global team of researchers, engineers, and designers passionate about AI, headquartered in Toronto with offices worldwide.

Global

  • Build LLM-powered software by designing prompt flows and orchestrations to ensure great performance with no false positives.
  • Architect and build a production-grade, testable, and maintainable AI-powered software stack.
  • Design experiments and evaluation frameworks for system performance testing at scale, conducting data analysis to draw conclusions.

XBOW builds an AI-powered platform that autonomously discovers, validates, and exploits vulnerabilities, providing proof-backed results in hours instead of weeks. Founded by the creator of GitHub Copilot and backed by Sequoia and Altimeter, the team includes world-class AI engineers and security researchers.

Global

  • Build and ship full-stack, AI-native products, internal platforms, prototypes, and production applications.
  • Design and implement agentic AI applications, LLM workflows, RAG systems, document-processing pipelines, and other applied AI solutions.
  • Translate ambiguous business, product, or client needs into scalable technical systems.

Infinity Constellation is a holding company building full-stack AI businesses, ranging from rapidly prototyping new products to embedding with companies to solve real-world problems. We are a focused network of builders who move quickly and create leverage across companies with AI at their core.

$100,000–$300,000/yr
US

  • Build the software backbone for autonomous foundation models, including multimodal data pipelines and training workflows.
  • Implement and iterate on LLM, VLM, and VLA architectures, owning model code paths and inference runners.
  • Deliver production-grade serving tooling for low-latency operation and own systems from architecture to iteration.

We're building the next generation of ground transportation with advanced physical AI to simplify freight challenges. Our stealth team, founded by engineers who scaled autonomous driving, is developing a new vehicle platform and focuses on creating reliable real-time autonomous systems.

$180,000–$200,000/yr
US

  • Design and build the company's AI-first platform architecture.
  • Develop agent-based systems, orchestration layers, and inference pipelines.
  • Build scalable data ingestion, transformation, and feature engineering pipelines.

Our client is an early-stage healthcare AI company leveraging AI, data science, and clinical expertise to generate actionable insights from health data. The small team works alongside experienced founders and world-class advisors, building an AI-native platform from scratch.

Global

  • Participate in research and development of AI-driven documentation solutions, owning features from concept to deployment.
  • Develop and maintain backend services and REST APIs using Python, FastAPI, and integrate AI capabilities with LLM orchestration frameworks.
  • Collaborate with cross-functional teams in Agile environment, contributing to frontend features with React and TypeScript, and ensuring system reliability and scalability.

Sigma Software is a technology company that develops AI-powered solutions for global customers, focusing on intelligent documentation platforms for the telecommunications domain. The company fosters a collaborative, flexible, and fully remote work environment with a focus on modern engineering practices.