Senior AI Inference Engineer

🔥 2 hours ago

🇺🇸 United States – Remote

⏰ Full Time

🟠 Senior

🤖 Artificial Intelligence

👻 Ghost score 15%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

StackYak

WebsiteLinkedInAll Job Openings

2 - 10 employees

🤖 Artificial Intelligence

☁️ SaaS

🤝 B2B

Artificial Intelligence • SaaS • B2B

StackYak is an AI infrastructure orchestration platform that combines inference, GPU compute, and private networking into a single control plane. It provides hosted inference and dedicated GPU capacity, built-in private connectivity, and observability so teams can move models from testing to production without stitching together separate vendors. The platform targets developers, enterprises, and MSPs/resellers, and plans to add storage and security layers in future releases.

📋 Description

• Determine how large language models should run on available GPU hardware and customer latency requirements • Select precision, quantization, parallelism, memory, batching, concurrency, topology, and serving runtime strategies • Benchmark, deploy, debug, tune, automate, and operate production inference systems • Own model placement and serving across single-GPU, multi-GPU, and multi-node environments • Select and manage serving runtimes including vLLM, SGLang, and TensorRT-LLM • Define concurrency limits, latency and TTFT targets, and throughput commitments based on evidence • Establish benchmarking methodology, harnesses, and reproducibility standards • Manage model loading, startup, health, upgrades, capacity, and failure handling in production • Handle multi-tenant and dedicated serving decisions • Investigate inference incidents across serving runtimes, GPUs, drivers, hosts, and networks • Turn inference expertise into auditable, tested, automated software • Help automate deployment and hardware, precision, runtime, concurrency, and latency-throughput decisions

🎯 Requirements

• Production experience running large language models under real load • Experience serving models across multiple GPUs and multiple nodes • Experience sizing models against hardware • Experience choosing quantization and precision strategies with quality and cost consequences • Experience tuning batching and concurrency • Experience measuring and defending latency, TTFT, throughput, utilization, and cost • Experience in NVIDIA and/or AMD inference environments • Production Python development for tooling and product • Experience debugging Linux and systems problems below the container boundary • Experience finding root causes of inference failures outside the serving runtime • Strong candidates may have experience with inference infrastructure, distributed inference, multi-GPU vendors/generations, serving-runtime benchmarking, deployment automation, schedulers, placement systems, or capacity planners • Willingness to build and test beyond an assigned roadmap • Must be comfortable with production responsibility, broad ownership, and ambiguity

🏖️ Benefits

• Competitive compensation plus meaningful equity • Remote and distributed work across time zones • Employment or contract engagement depending on location • Direct access to founders • Participation in technical decisions • Shared responsibility for production and on-call • Opportunity to work across design, implementation, debugging, and operations • High scope and ownership in an early-stage startup

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume