Staff Software Engineer, AI Inference Gateway

Job not on LinkedIn

🕒 September 19

🇺🇸 United States – Remote

💵 $180k - $220k / year

⏰ Full Time

🔴 Lead

🤖 AI Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 6%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Salad

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2018

🤖 Artificial Intelligence

💳 Fintech

💰 $17M Series A on 2022-03

Artificial Intelligence • Cloud Computing • Fintech

Salad is a cloud computing company that specializes in democratizing access to GPU resources for AI and machine learning applications. It offers SaladCloud, a platform that connects idle consumer GPUs to businesses in need of computational power, allowing for significant cost savings on cloud services. Salad enables users to deploy AI/ML models at scale efficiently and affordably, leveraging a vast network of distributed nodes worldwide.

📋 Description

• Own the AI Gateway system end to end, including request routing, streaming, batch/async job handling, and horizontal scaling across multiple gateway servers • Build and tune fleet-efficiency algorithms for node autoscaling, performance scoring, and eviction of underperformers • Operate and scale inference nodes running vLLM, llama.cpp, and similar servers • Design and maintain OpenTelemetry-based observability across the gateway and fleet • Maintain accurate pay-per-token and subscription billing integrations based on telemetry • Benchmark new models, quantizations, and GPU hardware for approximately 25% of working time • Work with product and marketing teams to decide what enters production • Collaborate on cross-system design and integrations with Salad engineering • Explain technical trade-offs to technical and non-technical teammates • Report directly to the CTO • Participate in shared after-hours support

🎯 Requirements

• Strong production Rust experience, ideally on high-throughput networked services • Hands-on experience running LLM inference servers (vLLM, llama.cpp, TGI, TensorRT-LLM, or similar) • Knowledge of KV cache behavior • Distributed systems fundamentals: load balancing, backpressure, failure handling, and unreliable nodes • Experience operating systems with metrics, tracing, and on-call responsibilities • Willingness to be on-call • Clear written and verbal communication • Nice to have: Pingora, Tokio, or proxy/gateway internals • Nice to have: experience with heterogeneous or consumer-grade GPU fleets, CUDA, or ROCm • Nice to have: quantization formats including GGUF, AWQ, GPTQ, and FP8 • Strong in at least two of Rust, LLM inference servers, and distributed systems

🏖️ Benefits

• Unlimited PTO • 75% of health insurance premiums covered for you and your dependents • Dental and vision coverage • 401(k) plan • Stock options • Company-provided computer • $500 WFH budget • Fully remote, with flexible hours

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume