Senior ML Systems Engineer, Inference

๐Ÿ•’ September 25

Where this job comes from

Posted on Runpod's careers page โ†—Runpod's careers page, not through a job board

First seen by Remote Rocketship September 25

Still open on Runpod's site. Checked 2 hours ago.

Apply Now
Logo of Runpod

Runpod

51 - 200 employees

Founded 2022

๐Ÿค– Artificial Intelligence

โ˜๏ธ SaaS

๐Ÿค B2B

๐Ÿ’ฐ $20M Seed on 2024-06

Artificial Intelligence โ€ข SaaS โ€ข B2B

Runpod is a cloud platform that provides on-demand GPU compute and managed infrastructure tailored for AI development and deployment. It offers GPU "Pods" across 31 global regions, serverless GPU endpoints for low-latency inference, multi-node GPU clusters for distributed training, and a hub for deploying open-source models and templates. Runpod emphasizes fast startup (sub-200ms cold starts), autoscaling from zero to thousands of workers, support for 30+ GPU SKUs, and tooling for the full AI lifecycle from experiment to production, targeting developers and enterprise AI teams.

๐Ÿ“‹ Description

โ€ข Define inference performance measurements, including throughput, time to first token, inter-token latency, and cost per token โ€ข Build tooling that makes performance measurements rigorous and repeatable โ€ข Profile and diagnose performance problems across the serving stack, from scheduling and memory management to kernels and interconnect โ€ข Improve serving efficiency for large, state-of-the-art models on single-node and multi-node GPU deployments โ€ข Turn findings into production-ready runtimes, configurations, and defaults โ€ข Collaborate with product and infrastructure teams to shape how inference is offered on Runpod โ€ข Monitor the fast-moving inference ecosystem and evaluate what to adopt, build, or contribute back โ€ข Trace serving engine/runtime bottlenecks and implement fixes when configuration tuning is insufficient โ€ข Own LLM serving performance end to end across models, hardware generations, and workloads

๐ŸŽฏ Requirements

โ€ข 5+ years of professional system engineering experience โ€ข Deep, hands-on experience with vLLM, SGLang, or a comparable serving engine in production or at serious benchmark scale โ€ข Strong software engineering skills in Python โ€ข Comfortable working in large, performance-critical codebases โ€ข Solid understanding of LLM inference performance, including batching, memory, parallelism, and latency-throughput trade-offs โ€ข Experience with inference optimization techniques such as quantization, speculative decoding, or distributed serving โ€ข Rigor in benchmarking and performance analysis โ€ข Comfort with GPU profiling tools โ€ข Ability to explain results clearly in writing and turn them into decisions โ€ข Eligible to work in the United States โ€ข Must not require employment visa sponsorship โ€ข Preferred: experience writing or tuning GPU kernels in CUDA or Triton โ€ข Preferred: contributions to inference or ML systems projects โ€ข Preferred: experience with multi-node GPU systems and high-speed networking โ€ข Preferred: experience at a company where inference cost and latency were core business metrics

๐Ÿ–๏ธ Benefits

โ€ข Meaningful equity in a fast-growing company; everyone on the team receives stock options โ€ข Generous medical, dental & vision plans โ€ข Flexible PTO โ€ข Remote work-first arrangement โ€ข $1,200 Home Office & Equipment Stipend โ€ข Passionate team on the cutting edge of AI infrastructure, with culture, learning, and ownership at the heart of how the company scales

Apply Now

๐Ÿ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

๐Ÿ•’ September 23

Slate Auto

201 - 500

๐Ÿš˜ Automotive

๐Ÿญ Manufacturing

๐Ÿš— Transport

AI/ML Engineer building production GenAI, computer vision, and data systems for Slateโ€™s affordable customizable vehicles. Applying AI across manufacturing, supply chain, and physical operations.

๐Ÿ•’ September 23

Press Ganey

1001 - 5000

๐Ÿ’ผ Consulting

๐Ÿ“ฆ Logistics

๐Ÿฅ Healthcare

ML Engineer II building production AI agents and LLM applications for Press Ganey, a healthcare experience measurement and analytics provider. Integrating advanced AI capabilities into Java/Spring enterprise platforms.

๐Ÿ•’ September 23

Press Ganey

1001 - 5000

๐Ÿ’ผ Consulting

๐Ÿ“ฆ Logistics

๐Ÿฅ Healthcare

Staff ML Engineer scaling production AI agents for Press Ganey, a healthcare experience measurement and analytics provider. Leading enterprise AI architecture across Java/Spring platforms and distributed systems.

๐Ÿ•’ September 22

Discord

501 - 1000

๐Ÿ’ผ Consulting

๐ŸŽฎ Gaming

๐ŸŒ Social Impact

Staff ML Engineer building ranking, targeting, and recommendation systems for Discordโ€™s Shop, Nitro, subscriptions, and gifting. Scaling ML platforms and revenue products for a global gaming community.

๐Ÿ•’ September 21

Bose Corporation

5001 - 10000

๐Ÿš˜ Automotive

๐Ÿญ Manufacturing

๐Ÿฝ๏ธ Food & Beverage

Principal Audio DSP/ML Engineer developing Bose audio algorithms for speech, acoustics, and consumer devices. Combining machine learning, deep learning, and digital signal processing for production hardware.

Developed byย Lior Neu-ner. I'd love to hear your feedback โ€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok