Senior ML Systems Engineer, Inference

🕒 September 25

🇺🇸 United States – Remote

💵 $150k - $220k / year

⏰ Full Time

🟠 Senior

🤖 Machine Learning Engineer

👻 Ghost score 10%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Runpod

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2022

🤖 Artificial Intelligence

☁️ SaaS

🤝 B2B

💰 $20M Seed on 2024-06

Artificial Intelligence • SaaS • B2B

Runpod is a cloud platform that provides on-demand GPU compute and managed infrastructure tailored for AI development and deployment. It offers GPU "Pods" across 31 global regions, serverless GPU endpoints for low-latency inference, multi-node GPU clusters for distributed training, and a hub for deploying open-source models and templates. Runpod emphasizes fast startup (sub-200ms cold starts), autoscaling from zero to thousands of workers, support for 30+ GPU SKUs, and tooling for the full AI lifecycle from experiment to production, targeting developers and enterprise AI teams.

📋 Description

• Define inference performance measurements, including throughput, time to first token, inter-token latency, and cost per token • Build tooling that makes performance measurements rigorous and repeatable • Profile and diagnose performance problems across the serving stack, from scheduling and memory management to kernels and interconnect • Improve serving efficiency for large, state-of-the-art models on single-node and multi-node GPU deployments • Turn findings into production-ready runtimes, configurations, and defaults • Collaborate with product and infrastructure teams to shape how inference is offered on Runpod • Monitor the fast-moving inference ecosystem and evaluate what to adopt, build, or contribute back • Trace serving engine/runtime bottlenecks and implement fixes when configuration tuning is insufficient • Own LLM serving performance end to end across models, hardware generations, and workloads

🎯 Requirements

• 5+ years of professional system engineering experience • Deep, hands-on experience with vLLM, SGLang, or a comparable serving engine in production or at serious benchmark scale • Strong software engineering skills in Python • Comfortable working in large, performance-critical codebases • Solid understanding of LLM inference performance, including batching, memory, parallelism, and latency-throughput trade-offs • Experience with inference optimization techniques such as quantization, speculative decoding, or distributed serving • Rigor in benchmarking and performance analysis • Comfort with GPU profiling tools • Ability to explain results clearly in writing and turn them into decisions • Eligible to work in the United States • Must not require employment visa sponsorship • Preferred: experience writing or tuning GPU kernels in CUDA or Triton • Preferred: contributions to inference or ML systems projects • Preferred: experience with multi-node GPU systems and high-speed networking • Preferred: experience at a company where inference cost and latency were core business metrics

🏖️ Benefits

• Meaningful equity in a fast-growing company; everyone on the team receives stock options • Generous medical, dental & vision plans • Flexible PTO • Remote work-first arrangement • $1,200 Home Office & Equipment Stipend • Passionate team on the cutting edge of AI infrastructure, with culture, learning, and ownership at the heart of how the company scales