Engineering Member – Pre-training and Inference Fault Tolerance

November 7

Apply Now

Receive Emails with Similar Jobs

poolside

WebsiteLinkedInAll Job Openings

Artificial Intelligence • Enterprise

poolside is a frontier AI lab and enterprise platform that builds and deploys foundation models, multi-agent systems, and developer-facing tools focused on automating complex software work. The company specializes in on-prem and VPC deployments, security-first integrations, governance, and connectors to enterprise data sources so organizations can run agents and models inside their own boundaries. Poolside embeds research and engineering with customers to deliver outcome ownership, risk controls, and measurable business impact while advancing toward AGI by starting in high-consequence software environments.

51 - 200 employees

Founded 2023

🤖 Artificial Intelligence

🏢 Enterprise

📋 Description

• Identify, study, and troubleshoot hardware problems during training at scale • Minimize the GPU idle time during faults, both operationally and strategically • Design and develop tools and add-ons to accelerate the training recovery • Improve the performance and reliability of checkpointing • Write high-quality Python (PyTorch), Cython, C/C++, CUDA API code

🎯 Requirements

• Understanding of Large Language Models (LLM) • Basic knowledge of Transformers • Knowledge of deep learning fundamentals • Strong engineering background • Programming experience • Linux API, Linux kernel • Strong algorithmic skills • Python with numpy, PyTorch, or Jax • C/C++ • NCCL • Use modern tools and are always looking to improve • Strong critical thinking and ability to question code quality policies when applicable • Distributed systems • Reliability • Observability • Fault-tolerance • K8s stack

🏖️ Benefits

• Fully remote work & flexible hours • 37 days/year of vacation & holidays • Health insurance allowance for you and dependents • Company-provided equipment • Wellbeing, always-be-learning and home office allowances • Frequent team get togethers • Great diverse & inclusive people-first culture