Source Job

Global

  • Own the inference stack end-to-end, including pipeline-parallel execution, placement and routing, transport, serving engine, and failure handling.
  • Invent and validate novel algorithms for fast inference on consumer hardware over the public internet, then put them into production.
  • Serve both RL training rollouts and final model serving in a permissionless, trustless setting.

Distributed Systems

20 jobs similar to Inference Stack Engineer

Jobs ranked by similarity.

$200,000–$300,000/yr
US

  • Bring new open-source models into production with a focus on latency and throughput.
  • Optimize inference systems including batching, KV cache, speculative decoding, and quantization.
  • Design, deploy, and operate heterogeneous clusters across infrastructure vendors to improve reliability and cost efficiency.

They are an early-stage AI infrastructure startup making large language model inference faster and more cost-efficient. The small team focuses on low-level performance engineering and AI-assisted workflows, with a strong emphasis on technical fundamentals.

$150,000–$220,000/yr
Global Unlimited PTO

  • Lead the effort to make Runpod the fastest and most cost-efficient place for LLM inference, owning performance end to end.
  • Profile and diagnose performance bottlenecks across the serving stack, from scheduling to kernels, and implement fixes.
  • Work closely with product and infrastructure teams to shape how inference is offered, turning improvements into production-ready defaults.

Runpod is the AI Developer Cloud, providing a platform for over one million developers to experiment, train, fine-tune, deploy, and scale AI. We're a small, remote-first team that takes ownership seriously, moves fast, and has processed more than 20 billion inference requests.

Global

  • Build the RL post-training stack end-to-end: rollout ingestion, reward computation, policy updates, and returning updated weights to the network.
  • Invent algorithms for asynchronous, high-latency, partially trusted generation, including staleness tolerance and off-policy corrections.
  • Ship the first decentralized post-trained models, building evals and taking the release from run to public artifact.

Pluralis Research is developing Protocol Learning, a decentralized approach to training and serving large models on consumer devices. We are a small, deeply technical team backed by Union Square Ventures, with a remote-first culture and team members spread globally.

Global

  • Distributed pretraining: Implement and optimize model-parallel training on heterogeneous GPUs under low-bandwidth, high-latency links.
  • Performance optimization: Apply techniques that reduce communication overhead while maintaining model convergence in challenging network environments.
  • Elasticity and fault tolerance: Build robust checkpointing and recovery so runs survive node churn as participants join and leave.

Pluralis Research is a decentralized AI research company developing Protocol Learning to train large models on consumer devices across the internet. The team is world-class and deeply technical, backed by tier-1 investors, with a remote-first culture.

Global

  • Own the threat model for decentralized training and inference, enumerating potential attacks and keeping it current.
  • Design and calibrate statistical verification methods with stated error rates across heterogeneous hardware.
  • Build and run the verification service in the inference path and take responsibility for its mistakes.

Pluralis Research is building Protocol Learning, a decentralized approach to training and serving large models on consumer-grade devices. They are a small, world-class team of ML researchers backed by Union Square Ventures, with a remote-first culture and an ideological commitment to trustless, sovereign AI.

$160,000–$190,000/yr
US

  • Own end-to-end ML systems from data and training to evaluation, inference, and deployment.
  • Build and evolve training, fine-tuning, and inference infrastructure for large models.
  • Partner with research and application engineering to turn model capabilities into product improvements.

Next Step Systems is an IT recruiting and staffing agency that connects companies with machine learning and technology talent. They specialize in placing candidates in remote and on-site roles across the United States and emphasize a fast-paced, results-oriented culture.

China

  • Operate and expand Telnyx's own B300 GPU fleet to maximize inference throughput per GPU-dollar.
  • Design and implement serverless inference for open-weight models and dedicated enterprise deployments.
  • Work upstream in open-source technologies like vLLM, SGLang, and Kubernetes.

Telnyx is an industry leader building the future of global connectivity through a private, multi-cloud IP network and edge APIs. The company is financially stable and profitable, with a global team and a focus on innovation and continuous learning.

Europe

  • Lead customer transitions from proof of concept to production, owning the customer journey and ensuring AI workloads run reliably and scale.
  • Optimize AI workload performance and reliability by analyzing architectures, improving inference efficiency, and troubleshooting complex issues.
  • Serve as a trusted technical partner, coordinating production incidents and driving continuous improvements across customer deployments.

Token Factory is an AI infrastructure company that helps businesses bring advanced AI applications from experimentation to reliable production at scale. Its culture is innovative, collaborative, and fast-moving, with a focus on autonomy and meaningful technical impact.

US Unlimited PTO

  • Build the component layer around our layout synthesis engine, including API contracts, services, and evaluation gates.
  • Turn model retraining into a one-command job with built-in benchmarks and readable results for the whole team.
  • Own latency and cost budgets for learned capabilities and partner with infrastructure engineers on MLOps and deployment.

Higharc is a VC-backed startup that is changing how new homes are designed and built using spatial AI and generative floor plan technology. The company is fully remote, has raised over $175M, and values flexibility, collaboration, and asynchronous deep work.

$140,000–$180,000/yr
Global

  • Build and operate ML infrastructure and platforms powering AI products.
  • Design systems for model training, evaluation, deployment, inference, and experimentation.
  • Optimize model serving and inference infrastructure for high-throughput, low-latency workloads.

US Unlimited PTO

  • Owns the architectural direction for Code Metal's AI platform and the team building it.
  • Builds production-grade inference serving, model gateways, agent harnesses, and context-engineering services.
  • Instruments the stack end to end and drives experimentation and observability for reliable AI systems.

Code Metal provides automated software engineering using AI and formal verification to produce provably correct code. Founded in 2023 and backed by Accel, Salesforce Ventures, and others, it serves defense and enterprise customers from offices in Boston and San Francisco.

US

  • Lead the end-to-end machine learning lifecycle, from data preparation and model training to evaluation, inference, and deployment.
  • Build and optimize scalable training, fine-tuning, and inference pipelines for large-scale models on GPU infrastructure.
  • Establish production best practices for model quality, reliability, safety, and measurable performance improvements.

This company builds production-grade machine learning systems and large-scale GPU-powered inference pipelines. It is a fully remote team that values technical rigor, collaboration, and hands-on engineering.

  • Own the architectural evolution of the operator foundation model, designing transformer variants for spatial domains and scaling distributed training beyond 45TB datasets.
  • Architect trillion-voxel inference systems that balance memory, compute, and communication with production-grade stability.
  • Ship expanded operator capabilities into production, increasing simulation throughput by 100x across global, multi-entity deployments.

Vinci builds operator intelligence infrastructure that modern hardware programs rely on, with a single foundation model deployed across industries on realistic production workloads. They are a Series A company with 45TB+ of physics data, running billion-voxel inference inside Tier-1 semiconductor environments.

Global

  • Drive down proving cost for zero-knowledge proofs of AI inference.
  • Build production-quality proof systems for frontier-scale models.
  • Own the path from research prototype to deployable verification tools.

SASH's Verification team builds and tests tools to verify international agreements about AI, prototyping new verification mechanisms and working with policymakers. They are a small, globally distributed team combining technical expertise with international AI policy experience.

$97,600–$139,000/yr
United States Canada

  • Build and maintain core infrastructure for Quora's ML platform, ensuring high availability, scalability, and performance.
  • Build and improve distributed systems serving ML models in production, from Large Recommendation Models to Large Language Models.
  • Work on GPU model serving, optimizing latency, throughput, and cost to support larger and more capable models.

Quora's mission is to grow the world's collective intelligence through two platforms: Quora for global knowledge sharing and Poe for AI agent collaboration. We are a remote-first company with passionate, collaborative, and high-performing global teams, rooted in transparency and experimentation.

Europe

  • Own network solutioning for k0rdent AI, designing GPU cluster interconnects and multi-tenant isolation.
  • Publish reference architectures and run proofs of concept on real hardware with customers.
  • Research emerging AI networking technologies and help shape the product roadmap.

Mirantis is a Kubernetes-native AI infrastructure company helping organizations build and operate scalable, secure infrastructure for modern AI workloads. It is committed to open standards, technical excellence, and a distributed, collaborative team culture.

US

  • Design high-performance training platform components including orchestration, observability, and performance tuning.
  • Deliver end-to-end ML pipelines from dataset curation to training, validation, and deployment.
  • Drive technical direction across ML Platform, Infrastructure, Autonomy, and Safety teams.

Stack is developing revolutionary AI and advanced autonomous systems for safer, more reliable, and efficient operations, focusing on autonomous trucking. The Stack team brings decades of experience in real-world systems and is committed to a culture of inclusion, entrepreneurship, and innovation.

Global

  • Build the Learning Gym, a sandboxed environment for training and evaluating AI agents on real Niural workflows with ground-truth reward signals.
  • Engineer verifiers, run reinforcement learning loops, and establish honest baselines to prove agent improvements generalize.
  • Write first-author research papers and internal technical reports, and represent the work externally through preprints and talks.

Niural is a global payroll, Employer of Record (EOR), Agent of Record (AOR), and Contractor Management platform. It is a relentlessly ambitious team focused on building foundational internet infrastructure for the AI era.

Europe

  • Own end-to-end technical strategy for major areas of the Voyager SDK’s application layer, including model deployment and pipeline development.
  • Set technical direction for image processing, create reusable architectures, and partner with the compiler team to resolve compilation issues.
  • Represent the Applications team’s capabilities to business stakeholders, mentor engineers, and drive documentation standards.

Axelera AI is a deep-tech company creating a next-generation AI platform to advance humanity. With $450 million raised and 250+ employees including 60+ PhDs, they have a strong business pipeline exceeding $100 million and are committed to innovation.

North America

  • Design and build systems that improve the efficiency of ML training and inference workloads.
  • Develop tooling and benchmarking frameworks to help ML engineers debug, profile, optimize, and monitor model performance.
  • Partner with ML teams to optimize distributed training, reduce costs, and drive technical strategy for scalability and reliability.

Reddit is a community of communities where users share interests and authentic conversations. With 100,000+ communities and ~130 million daily visitors, it has a flexible-first culture supporting remote work.