Senior ML Engineer – Inference Stack Optimization

🔥 8 minutes ago

🌐 Germany, Austria, +6 more countries – Remote

🇩🇪 Germany – Remote 🇦🇹 Austria – Remote 🇫🇷 France – Remote 🇮🇹 Italy – Remote 🇳🇱 Netherlands – Remote 🇵🇱 Poland – Remote 🇪🇸 Spain – Remote 🇬🇧 United Kingdom – Remote

⏰ Full Time

🟠 Senior

🤖 Machine Learning Engineer

👻 Ghost score 12%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Cast AI

WebsiteLinkedInAll Job Openings

201 - 500 employees

Founded 2019

☁️ SaaS

🤖 Artificial Intelligence

🤝 B2B

SaaS • Artificial Intelligence • B2B

Cast AI is a SaaS platform that automates Kubernetes application performance, infrastructure, and cost optimization for cloud-native teams. It connects to EKS/AKS/GKE and on-prem clusters with no code changes, uses predictive models and runtime signals to rightsize pods, scale nodes, optimize GPUs and Spot instances, and run approved self-healing runbooks to maintain reliability and reduce cloud spend. Cast AI targets platform, SRE, and FinOps teams and supports AI/LLM workloads, GPU optimization, and real-time workload placement and cost intelligence.

📋 Description

• Optimize inference-stack throughput, latency, cold starts, and cost per token • Tune continuous batching, chunked prefill, speculative decoding, CUDA-graph capture, and schedulers across GPU fleets • Optimize time to first token and inter-token latency by profiling compute, memory bandwidth, scheduling, storage, and network bottlenecks • Improve KV-cache utilization through hierarchical and paged attention, prefix caching, eviction policies, and quantized KV • Reduce cold starts using O_DIRECT and fastsafetensors-class weight loaders, JIT and compile caches, and shared-filesystem read paths • Quantize weights, activations, and KV using INT8/INT4 and FP8/FP4 while preserving quality on real workloads • Scale distributed inference across nodes through network-aware placement, prefill/decode disaggregation, and non-bottlenecking checkpoint strategies • Set technical direction by choosing benchmarks, technologies to adopt, and systems to build; support decisions with writeups and reproducible experiments

🎯 Requirements

• 5+ years of experience building real ML systems • Portfolio demonstrating depth in inference or training infrastructure, not just model training notebooks • Production-grade Python for services • Real-world experience with vLLM, SGLang, or TensorRT-LLM • Working understanding of engine performance on different GPUs • Experience measuring quantization regressions • Experience with multi-GPU and multi-node setups, including tensor and expert parallelism and collective communication • Knowledge of page caches, O_DIRECT, network filesystems, and terabyte-scale reads • Ability to instrument systems before optimizing and distinguish genuine improvements from benchmark artifacts • Experience with LLM routing, RouterBench-class evaluations, cost/quality/latency Pareto fronts, calibration, abstention, conformal methods, online A/B loops, and drift monitoring is a nice-to-have • Must complete the hiring process, including recruiter screening, hiring manager interview, technical system design interview, live coding, and executive culture check • Background check may be conducted at the final recruitment stage • Cast AI does not provide visa sponsorship or work permits

🏖️ Benefits

• Competitive salary (depending on the level of experience) • Enjoy a flexible, remote-first global environment • Equity options • 10% of work time for personal projects or self-improvement • Learning budget for professional and personal development, including international conferences and courses • Annual hackathon • Team-building budget and company events • Equipment budget • Extra days off

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume