Staff Software Engineer, LLM Inference Platform

🕒 October 29

🏄 California – Remote

🏄 California – Remote

💵 $300k - $600k / year

⏰ Full Time

🔴 Lead

🗣️ LLM Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 0%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Open for 16 days.
• Jobs at this company usually fill in 53 days.

Apply Now

Check your resume score

Customize resume + cover letter

Find Similar Remote Jobs

☆ Save☑️ Mark as applied❌ Hide

Report problem

Zenlayer

WebsiteLinkedInAll Job Openings

201 - 500 employees

Founded 2014

💼 Consulting

📦 Logistics

📣 Marketing

💰 Series C on 2022-01

Consulting • Logistics • Marketing

Zenlayer is a hyperconnected cloud service provider focused on deploying and running applications close to users. With over 50 countries covered and 300 edge nodes, Zenlayer offers solutions like on-demand dedicated servers, scalable virtual servers, and edge data center colocations. Their services include cloud connectivity, network performance optimization, and multi-cloud services, specifically designed to enhance application performance and reduce latency across emerging markets. Zenlayer emphasizes intelligent routing and private global networking to improve user satisfaction and network efficiency. They serve industries such as gaming, blockchain, VPN, and cloud services worldwide.

📋 Description

• Own end-to-end responsibility for the platform's Token Performance • Continuously reduce cost per million tokens and increase per-GPU Goodput while meeting TTFT/TPOT SLOs • Prove the gain from every optimization with reproducible data • Establish a unified Token Performance metrics framework covering TTFT, TPOT, Goodput, GPU utilization, and per-token cost • Build load-testing benchmarks reflecting high-concurrency, long-context, and multi-turn Agent production workloads • Perform bottleneck attribution for DeepSeek, Kimi, GLM, Qwen, and MiniMax across compute, memory bandwidth, communication, scheduling, and queuing • Customize and contribute to vLLM, SGLang, and TensorRT-LLM on scheduler, Continuous Batching, Chunked Prefill, CUDA Graph, and KV block management paths • Define responsibility boundaries between inference engines and orchestration layers such as Dynamo and llm-d • Reduce tail latency through request routing and load balancing • Lead Prefill-Decode disaggregation design, including KV transfer via NIXL/RDMA/UCX, transfer-compute overlap, and phase-aware autoscaling • Define failure domains and recovery semantics for cross-instance KV transfer • Design hybrid TP/PP/DP/EP parallelism strategies and quantify EP/TP trade-offs • Address expert load imbalance and All-to-All communication overhead and drive Wide-EP into production • Write and tune CUDA and ROCm/HIP operators, including memory access optimization, kernel fusion, and Attention/MoE/communication kernels • Build a quantization-accuracy-throughput evaluation loop for FP8, FP4, INT4, AWQ, and GPTQ across engines and hardware

🎯 Requirements

• First-principles understanding of Transformer inference: the compute vs. memory-bound nature of Prefill and Decode, the KV Cache memory model, the differences among MHA/GQA/MLA, and how MoE routing constrains parallelism • Source-level, in-depth experience with at least one mainstream inference engine, with real cases of identifying and fixing bottlenecks in scheduling, kernel selection, or KV management • CUDA kernel tuning skills (Nsight profiling, memory optimization, fusion, CUDA Graph) with hands-on practical experience • Bachelor’s degree in Computer Science, Electrical/Computer Engineering, or related field (HPC systems, ML Systems) • Minimum 6 years in systems/Performance Engineering, with at least 2-3 years directly on LLM or GPU performance work • Experience with Merged PRs in vLLM preferred, but not required • Experience operating at cluster scale (hundreds to thousands of GPUs) preferred, but not required

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume