Senior Machine Learning Engineer, AI Safety Evaluations

🔥 4 minutes ago

🏄 California – Remote

🏄 California – Remote

💵 $184k - $356.5k / year

⏰ Full Time

🟠 Senior

🤖 AI Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 1%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

NVIDIA

WebsiteLinkedInAll Job Openings

10,000+ employees

Founded 1993

🏥 Healthcare

🏭 Manufacturing

🤖 Artificial Intelligence

Healthcare • Manufacturing • Artificial Intelligence

NVIDIA is a leading technology company specializing in accelerated computing and artificial intelligence. NVIDIA pioneers advancements in graphical processing units (GPUs), cloud computing, data centers, and virtual reality, with a focus on gaming, automotive, healthcare, and robotics industries. The company's innovations, such as NVIDIA Omniverse, transform traditional digital processes by enabling high-fidelity simulations and rendering tasks. Their applications span various industries, from autonomous vehicles using NVIDIA DRIVE to healthcare solutions with NVIDIA Clara, and AI-driven analytics and workflows.

📋 Description

• Design and maintain safety benchmarks for large language and multimodal models • Translate risk scenarios into test cases, datasets, and scoring rubrics • Define metrics and protocols assessing safety alongside model usefulness • Use baselines, uncertainty estimates, and failure analysis to distinguish meaningful improvements from noise • Build reusable Python pipelines with versioned datasets, model configurations, and experiment tracking • Develop automated red-teaming tools to discover failure modes and expand evaluation coverage • Incorporate human review to validate automated scoring and provide additional judgment • Investigate model failures with researchers and document actionable findings • Develop regression tests to assess whether model changes resolve identified issues • Partner with platform engineers to store and aggregate evaluation results • Communicate findings, limitations, and emerging risks to researchers and decision-makers

🎯 Requirements

• Bachelor’s degree or equivalent experience • 6+ years in machine learning engineering or evaluation • Experience developing benchmarks, metrics, and statistically sound comparisons for ML systems • Ability to translate ambiguous evaluation questions into testable experiments • Strong Python skills • Experience building maintainable shared tools, data pipelines, and experiment-tracking workflows • Ability to assess dataset quality and identify limitations in metrics and scoring methods • Ability to support conclusions with reproducible evidence • Ability to clearly explain findings and tradeoffs to researchers, engineering partners, and executives • Experience evaluating LLMs, multimodal models, or agentic systems through adversarial testing or automated red-teaming • Contributions to public benchmarks or evaluation frameworks • Publication track record and interest in continuing to publish at top tier venues • Experience validating model-based evaluators against human judgments or designing human-in-the-loop evaluation workflows • Strong model evaluation skills and commitment to sound statistical practices, including confidence intervals

🏖️ Benefits

• Equity • Benefits

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume