Design and build automated evaluation pipelines for AI agents.
Develop testing frameworks for reasoning, multi-turn conversations, and adversarial scenarios.
Own the evaluation architecture and integrate observability tools in a fully remote team.
A company building production-grade AI agents and systems. They operate in a fully remote, globally distributed environment with a culture focused on innovation, autonomy, and continuous improvement.
Design and build challenging evaluations and executable environments for coding and long-horizon agents.
Develop scoring logic and verifiers to detect reward hacks and trace failures through model outputs and harness code.
Build and scale RL data pipelines for generating and curating challenging coding and agentic tasks.
Prime Intellect is building the open superintelligence stack, providing infrastructure for post-training frontier AI models and open-source models trained for long-horizon tasks. The company has raised $150M from top investors and is a mission-driven team passionate about accelerating science and AI.
Building multi-agent systems and implementing agent-based workflows and AI automation solutions.
Developing and optimizing prompt engineering strategies, including evaluation and iteration.
Integrating LLMs and AI models into cloud-based applications and APIs while researching new AI tools.
Software Mind develops impactful software solutions for global companies, partnering with tech giants and unicorns on transformative projects. The company builds cross-functional engineering teams and embraces a culture of openness, respect, grit, and cooperation.
Build LLM-based agents on the platform's scaffolding, integrating tool calls, internal APIs, and guardrails.
Take agents to production on AWS with containers, CI/CD, secrets, permissions, and security controls.
Define and run evals, monitor with Langfuse, and document runbooks for independent operation.
Muttdata builds innovative Data Products and Machine Learning solutions to help companies solve complex business challenges. It is a fast-growing, remote-first startup that values collaboration, continuous learning, and a positive, ownership-driven culture.
Interact with AI models across a wide range of prompts and topics.
Compare AI-generated responses side by side to assess quality.
Write clear, well-reasoned explanations supporting your evaluations.
Anyone AI creates high-quality training and evaluation data for frontier AI models. They work with remote independent contractors, focusing on expert human evaluation to improve AI accuracy and reliability.
Develop and deploy cutting-edge AI/ML solutions to enhance the platform and improve student learning experiences.
Design, develop, and optimize LLM-powered agentic systems and APIs for real product use cases.
Collaborate with senior engineers and contribute to evaluation frameworks and MLOps pipelines.
Interview Kickstart specializes in interview preparation and career transitions into high-demand tech fields like AI, ML, and Data Science. Over 17,000 tech professionals have been guided by current and former hiring managers to land coveted positions at companies like Google and Amazon.
Lead the engineering strategy and execution for evaluations of AI agents, owning the core evaluation platform.
Design scalable evaluation infrastructure, APIs, workflows, and production systems across software categories.
Mentor and develop a team of engineers as the technical authority on agentic evaluation.
This company focuses on building credible, scalable evaluations of AI agents from software vendors. It operates as a fully remote, inclusive team with a flexible culture and a focus on professional growth.
Study how engineers and customers use Archie in the field, identify where it succeeds or fails, and turn observations into actionable evaluations.
Recreate real-world engineering tasks and failure modes in repeatable environments for development teams.
Build and refine evaluation methodologies, including automated judges and human evaluation processes, to align with expert judgment.
P-1 AI is building Archie, an AI engineer agent for the physical world that works alongside human engineering teams. The company recently raised a $50 million Series A led by NEA and is driven by the mission of building superintelligence for engineering.
Design and improve AI systems for extracting structured reasoning signals from technical interviews.
Build LLM-powered pipelines for semantic analysis, behavioral feature extraction, and evaluation.
Develop scoring, calibration, and benchmarking methods grounded in measurable evidence.
APADCode is an early-stage AI startup building an AI-powered technical interview platform that evaluates how candidates reason and solve problems. The company is a small, early-stage team with a culture of high ownership, direct collaboration, and minimal bureaucracy.
Build tools for understanding traces, evaluating agent behavior, and turning results into improvements.
Run experiments on Mastra's primitives to find where they fall short and improve configuration and behavior.
Help turn findings into capabilities developers can use in their own applications.
Mastra is the open-source TypeScript framework for building AI agents. We're a small, fully remote team backed by Y Combinator and Spark Capital with $35M raised.
Build and maintain AI agents using large language models to complete multi-step tasks within Cayuse products.
Develop and extend MCP servers and integrate AI features into existing applications through platform APIs.
Write unit and integration tests, participate in code reviews, and ship code through CI/CD pipelines.
Cayuse provides integrated technology solutions that empower organizations to conduct globally connected research, advancing impact in science and discovery. The company is a team of highly skilled, multidisciplinary professionals building a remote-first culture with a focus on customer and employee values.
Evaluate AI model responses across diverse topics and use cases against established quality standards.
Write, test, and refine system prompts to influence model behavior and performance.
Identify and document model failures and provide clear, actionable feedback to product teams.
This company specializes in evaluating and improving advanced AI systems through content evaluation and prompt engineering. It operates as a partner company managing applications for this role, focusing on quality and consistency in AI behavior.
Build AI features inside real clinical workflows, like checking notes against payer and compliance requirements before finalization.
Design evals to prove those features are right and keep them honest as models change.
Prototype AI capabilities, including voice agents, and take the best from prototype to production.
Ease builds the AI-native operating system for behavioral health, replacing disconnected tools with one platform for CRM, EHR, and revenue cycle management. Backed by a $41M Series A, this rapidly growing NYC team has no product managers—engineers talk to customers and ship in days.
Lead the engineering foundation behind reliable, measurable, and scalable AI-powered features.
Build evaluation frameworks, observability tooling, and diagnostic infrastructure for AI agents in production.
Combine hands-on software engineering with technical leadership and people management.
The company builds AI-powered products with a focus on reliability, scalability, and evaluation infrastructure. It operates as a fully remote, globally distributed team with an ownership-oriented culture.
Implement embeddings, vector storage, and retrieval across a large, provenance-tracked evidence base.
Integrate language models so generated text is bound to cited source records, and test for citation failures.
Own prompt and output-schema design, model packaging, versioning, serving, and rollback.
DEFCON AI is an insights company that leverages artificial intelligence, mathematical optimization, data analytics, and software engineering to build resilient optimization systems for complex operations. The company offers a fully remote, results-based environment with a focus on production systems and a strong benefits package.
Build AI-powered agents and tools end-to-end: identify problems with clients, prototype quickly, evaluate rigorously, and ship to production.
Design and implement LLM integrations including prompt engineering, function calling, structured outputs, and retrieval-augmented workflows.
Work directly with clients to understand workflows, validate AI behavior, and iterate toward production quality.
Addepar is a global data and AI platform empowering investment professionals to turn complex financial information into actionable intelligence. With over 1,500 firms in 60 countries and nearly $10 trillion in assets on the platform, Addepar fosters a culture of ownership, collaboration, and continuous learning.
Define end-to-end requirements for AI capabilities, from model behavior to user experience.
Translate model capabilities and technical constraints into product decisions.
Work closely with ML and engineering teams on system design and iteration.
They develop AI-powered email applications and work at the intersection of user needs and model capability. The company is fast-moving and collaborative with a strong focus on technical ownership and innovation.
Design and deploy AI agents that automate sales, operations, and support workflows.
Own technical delivery end-to-end, from architecture to production and post-launch improvements.
Work directly with clients to understand workflows and explain technical tradeoffs.
Stello is an AI transformation firm that partners with mid-market and enterprise companies to automate repetitive work with AI agents. They work across sales, operations, and customer support, deploying solutions inside existing client tools.