Cluster Engineer

Job not on LinkedIn

🔥 0 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

STN Incorporated

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2016

🏢 Enterprise

🔒 Cybersecurity

🔧 Hardware

Enterprise • Cybersecurity • Hardware

STN Incorporated is an enterprise-grade managed IT and cloud infrastructure provider that delivers secure, audit-ready infrastructure for business-critical systems and demanding AI workloads. STN operates a managed operating model offering private CPU clouds, GPU One AI infrastructure, secure networking and storage, and 24/7 human support with high uptime SLAs. Their services include managed infrastructure and cloud operations, cybersecurity operations and incident response, compliance and risk management (SOC 2 Type II, HIPAA-ready), backup and recovery, and enterprise technology procurement and lifecycle management. STN serves enterprises, high-growth SaaS companies, AI builders and model developers, robotics/physical AI firms, and regulated industries such as healthcare.

📋 Description

• Design, deploy, and optimize multi-node GPU clusters for AI training and inference workloads • Tune distributed training environments to maximize GPU utilization, throughput, and scaling efficiency • Optimize inference clusters for token generation throughput, low latency, and high GPU utilization • Build and support production AI infrastructure running hundreds to thousands of GPUs • Analyze and eliminate performance bottlenecks across compute, networking, storage, and software layers • Perform NCCL benchmarking, analysis, and tuning for collective communication performance • Design and optimize GPU networking using InfiniBand or RoCE v2, including RDMA, congestion management, topology awareness, and QoS • Configure and tune distributed AI software stacks including PyTorch, NCCL, CUDA, UCX, MPI, Slurm, and Pyxis/Enroot • Optimize GPU scheduling and resource allocation for training and inference environments • Develop benchmarking and validation processes for hardware, firmware, drivers, and software releases • Identify performance regressions and troubleshoot distributed training issues at scale • Optimize storage architectures for checkpointing, dataset streaming, and high-performance parallel I/O • Work closely with ML engineers to improve training scalability and inference efficiency • Create automation to deploy, validate, benchmark, and monitor GPU clusters • Evaluate emerging AI infrastructure technologies and recommend platform architecture improvements

🎯 Requirements

• 7+ years designing or operating large-scale Linux infrastructure • 5+ years supporting production GPU clusters for AI or HPC workloads • Experience building multi-node GPU training environments from the ground up • Deep expertise with distributed PyTorch training • Extensive experience troubleshooting and optimizing NCCL communications • Understanding of AllReduce, ReduceScatter, AllGather, Broadcast, and point-to-point communications • Experience benchmarking distributed training with nccl-tests, NVIDIA DCGM, and Nsight Systems; MLPerf preferred • Understanding of GPU memory management, KV Cache, activation checkpointing, tensor parallelism, pipeline parallelism, and data parallelism • Experience optimizing LLM inference throughput, including tokens/sec, batch sizing, continuous batching, KV cache, and memory bandwidth • Experience tuning CUDA, NCCL, UCX, and MPI • Expert-level Linux systems administration skills • Experience with Slurm • Experience using Pyxis and Enroot for containerized GPU workloads • Strong Python and Bash scripting skills • Strong understanding of InfiniBand, RoCE v2, RDMA, GPUDirect RDMA, GPUDirect Storage, UCX, MPI, network topology optimization, congestion control, QoS, ECN/PFC, and high-speed Ethernet • Experience designing or tuning AI storage, including parallel file systems, distributed storage, object storage, NVMe, checkpoint optimization, dataset staging, storage bandwidth, and metadata performance • Experience with NCCL benchmarking, multi-node scaling analysis, GPU utilization optimization, communication/computation overlap, NUMA optimization, CPU affinity, PCIe topology, GPU topology, memory bandwidth analysis, and end-to-end performance profiling • Preferred: Kubernetes, NVIDIA GPU Operator, Kubernetes batch scheduling, vLLM/TensorRT-LLM/SGLang, NVIDIA DGX SuperPOD or similar, MLPerf, Prometheus/Grafana/DCGM Exporter, Ansible/Terraform, and AWS/Azure/GCP GPU environments