HPC Operations Engineer

November 9, 2023

Lambda

WebsiteLinkedInAll Job Openings

Designing the world's most advanced GPU systems for Deep Learning.

Deep Learning • Machine Learning • Artificial Intelligence

51 - 200

💰 $39.7M Venture Round on 2022-11

Description

• Remotely deploy and configure large-scale HPC clusters for AI workloads (up to many thousands of nodes) • Remotely install and configure operating systems, firmware, software, and networking on HPC clusters both manually and using automation tools • Troubleshoot and resolve HPC cluster issues working closely with physical deployment teams on-site • Provide context and details to an automation team to further automate the deployment process • Provide clear and detailed requirements back to HPC design team on gaps and improvement areas, specifically in the areas of simplification, stability, and operational efficiency • Contribute to the creation and maintenance of Standard Operating Procedures • Provide regular and well-communicated updates to project leads throughout each deployment • Stay up-to-date on the latest HPC/AI technologies and best practices

Requirements

• Good understanding of HPC/AI architecture, operating systems, firmware, software, and networking • 3+ years of experience in deploying and configuring HPC clusters for AI workloads • Innate attention to detail • Familiarity with Bright Cluster Manager or similar cluster management tools • Experience in configuring and troubleshooting: SFP+ fiber, InfiniBand (IB), and 100 GbE network fabrics, Ethernet, switching, power infrastructure, GPU direct, RDMA, NCCL, Horovod environments, Linux-based compute nodes, firmware updates, driver installation, SLURM, Kubernetes, or other job scheduling systems • Ability to work well under deadlines and structured project plans also knowing when and how to ask for changes to project timelines • Solid problem-solving and troubleshooting skills • Flexibility to travel to North American data centers when needed • Ability to work independently and as part of a team • Nice to Have: experience with machine learning and deep learning frameworks (PyTorch, TensorFlow) and benchmarking tools (DeepSpeed, MLPerf), experience with containerization technologies (Docker, Kubernetes), experience working with GPU acceleration, virtualization, and cloud computing technologies • Situational awareness in customer situations, employing diplomacy and tact • Bachelor's degree in EE, CS, Physics, Mathematics, or equivalent work experience

Benefits

• Generous cash & equity compensation • Health, dental, and vision coverage for you and your dependents • Commuter/Work from home stipends • 401k Plan • Flexible Paid Time Off Plan • Salary Range: $120,000-$160,000, depending on qualifications