Senior AI Inference Engineer

🔥 2 hours ago

🌐 United States, Israel – Remote

🇺🇸 United States – Remote 🇮🇱 Israel – Remote

⏰ Full Time

🟠 Senior

🤖 Artificial Intelligence

👻 Ghost score 15%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

StackYak

WebsiteLinkedInAll Job Openings

2 - 10 employees

🤖 Artificial Intelligence

☁️ SaaS

🤝 B2B

Artificial Intelligence • SaaS • B2B

StackYak is an AI infrastructure orchestration platform that combines inference, GPU compute, and private networking into a single control plane. It provides hosted inference and dedicated GPU capacity, built-in private connectivity, and observability so teams can move models from testing to production without stitching together separate vendors. The platform targets developers, enterprises, and MSPs/resellers, and plans to add storage and security layers in future releases.

📋 Description

• Determine model placement, parallelism, precision, and memory strategies across single-GPU, multi-GPU, and multi-node deployments • Select and evaluate serving runtimes including vLLM, SGLang, TensorRT-LLM, and successors • Establish and defend concurrency limits, latency and TTFT targets, and throughput commitments • Develop benchmarking methodology, harnesses, and reproducible measurement practices • Own production model lifecycle including loading, startup, health, upgrades, capacity, and failure handling • Design multi-tenant and dedicated serving approaches • Investigate and resolve inference incidents involving serving runtimes, GPUs, drivers, hosts, and networks • Convert inference expertise into auditable, tested, and automated software • Deploy, benchmark, debug, tune, automate, and operate real inference systems in production • Help automate decisions about hardware fit, precision, model splitting, serving runtime, concurrency, and latency-throughput tradeoffs • Participate in technical decisions and share production/on-call responsibility

🎯 Requirements

• Proven production experience running large language models under real load • Experience serving models across multiple GPUs and multiple nodes • Experience sizing models against hardware • Experience selecting quantization and precision strategies with quality and cost implications • Experience tuning batching and concurrency • Experience measuring and defending latency, TTFT, throughput, utilization, and cost metrics • Experience with NVIDIA and/or AMD inference environments • Production Python development • Linux and systems debugging below the container boundary • Experience diagnosing inference failures outside the serving runtime • Understanding of distributed inference and multi-node networking • Experience benchmarking and comparing serving runtimes • Experience automating deployment decisions, or building schedulers, placement systems, capacity planners, or similar infrastructure • Ability to work across design, implementation, debugging, and operations • Participation in production on-call • Willingness to work in a fast-paced, ambiguous early-stage startup environment

🏖️ Benefits

• Meaningful equity • Remote and distributed work across time zones • Employment or contract engagement depending on location • Direct access to founders • Shared responsibility for production and on-call • Participation in technical decisions

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume