Model Serving Engineer

Job not on LinkedIn

🕒 April 2

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Fundamental

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2024

🤖 Artificial Intelligence

🏢 Enterprise

☁️ SaaS

Artificial Intelligence • Enterprise • SaaS

Fundamental is an enterprise AI company that builds large tabular models (LTMs) such as NEXUS, pre-trained on billions of tables to detect patterns and predict outcomes from structured data. The company offers an enterprise-grade predictive analytics platform that can be deployed with minimal code or integrated deeply with cloud partners like AWS, emphasizing privacy, security, and scalability. Born from academic research and backed by major investors, Fundamental targets large organizations seeking to extract foresight from their databases and deploy predictive models at cloud scale.

📋 Description

• Design, build, and maintain production model serving infrastructure using Triton Inference Server as the primary framework • Implement and optimize inference pipelines including custom backends, dynamic batching strategies, and model ensemble configurations in Triton • Optimize Python inference code for performance, with a strong focus on GIL contention, multi-threading, and concurrency patterns • Tune throughput and latency across the full serving stack, batching policies, thread pool sizing, model instance groups, and memory layout • Work closely with the research team to understand new model architectures at a computational level, batching behavior, dynamic shapes, memory access patterns etc • Own the full resource observability and control loop for production inference - instrument GPU memory, CPU, batch queue depth, and latency metrics, and actively tune model instance groups, concurrency limits, memory budgets, and batching configuration in response to observed behavior • Evaluate and integrate alternative inference frameworks and runtimes as the model ecosystem evolves • Contribute to GPU utilization improvements and resource efficiency across the serving fleet

🎯 Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field (or equivalent practical experience) • 5+ years of experience in model serving, ML infrastructure, or a closely related backend engineering role • Deep, production-level experience with Triton Inference Server, including custom Python backends, batching configuration, and model repository management • Expert-level Python skills with a thorough understanding of the GIL, multi-threading, multiprocessing, and async concurrency patterns • Strong understanding of neural network inference mechanics, forward passes, batching strategies, memory management, and numerical precision tradeoffs • Hands-on experience with other inference frameworks (TorchServe, TensorFlow Serving, ONNX Runtime, vLLM, etc.) and the ability to evaluate tradeoffs between them • Experience profiling and optimizing inference code for latency and throughput at production scale

🏖️ Benefits

• Competitive compensation with salary and equity • Comprehensive health coverage, including medical, dental, vision, and 401K • Paid parental leave for all new parents, inclusive of adoptive and surrogate journeys • Relocation support for employees moving to join the team in one of our office locations • A mission-driven, low-ego culture that values diversity of thought, ownership, and bias toward action