ML Platform Engineer, Distributed Training

🔥 13 hours ago

🇺🇸 United States – Remote

💵 $159.3k - $219.2k / year

⏰ Full Time

🟡 Mid-level

🟠 Senior

🏗️ Platform Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 1%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Egen

WebsiteLinkedInAll Job Openings

501 - 1000 employees

Founded 2000

💼 Consulting

📦 Logistics

📣 Marketing

Consulting • Logistics • Marketing

Egen is a company that specializes in engineering innovative solutions using platforms, data, and generative AI. They focus on harnessing the potential of data to empower organizations and individuals, providing services across various sectors including communications, healthcare, and public sector solutions. Their customizable platforms address critical challenges in urban management, customer engagement, clinical care, and operational excellence, aiming to drive impactful outcomes for clients.

📋 Description

• Work with the client's infrastructure team on the GKE training platform, including GPU node pools, job queueing and scheduling, quota, and capacity • Diagnose and fix platform failures at the root cause • Tune multi-node GPU networking, including RDMA/GPUDirect and NCCL, for training throughput • Run, scale, and debug multi-node distributed training with PyTorch and Ray Train • Profile jobs and remove bottlenecks across compute, memory, input pipelines, and interconnect • Design and run performance benchmarks for the cluster and training stack • Build and maintain model evaluation harnesses • Instrument training and the platform with telemetry and monitoring • Maintain end-to-end pipeline tests against real user workloads • Proactively identify issues, address gaps, propose solutions, communicate status and risks, and support scientists using the platform • Explain and defend design decisions to client engineers and scientists • Adapt effectively as client priorities and team structure change • Develop document intelligence and retrieval-augmented generation (RAG) features from prototype through scalable production services • Translate client business needs into high-performance AI architectures

🎯 Requirements

• 5+ years of engineering experience, with recent hands-on work on GPU training infrastructure and large-scale training • Mastery of Python and shell scripting • Extensive experience with Google Gemini, GPT-4, or LLaMA • Deep knowledge of prompt engineering and fine-tuning • Expertise in vector databases, including Vertex AI Vector Search and pgvector • Experience implementing semantic search • Hands-on experience with Google Cloud (Vertex AI) and scalable software architectures • Proficiency in LangChain, LlamaIndex, or similar orchestration layers • Strong Python software engineering, including production-quality code, tests, and code review • Experience with Kubernetes on Google Cloud, GPU node pools, job queueing, and capacity/quota management • Personal experience running and debugging multi-node distributed training jobs with PyTorch and Ray Train • Experience with GPUDirect/RDMA, NCCL tuning, and multi-node GB200 or H200 topology • Experience designing performance benchmarks and model evaluations • Ability to explain system operation and rationale without relying on AI coding tools • Track record of visible ownership, communication, and accountability on a demanding team • Healthcare or life-sciences data experience and HIPAA experience are nice to have • Observability and telemetry for training platforms are nice to have

🏖️ Benefits

• Comprehensive Health Insurance • Paid Leave (Vacation/PTO) • Paid Holidays • Sick Leave • Parental Leave • Bereavement Leave • 401(k) Employer Match • Employee Referral Bonuses