Member of Technical Staff, Evals

🔥 15 hours ago

🏄 California – Remote

🏄 California – Remote

💵 $200k - $550k / year

⏰ Full Time

🔴 Lead

🖥 Software Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 0%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Open for 1 day.

Apply Now

Find Similar Remote Jobs

Magic

WebsiteLinkedInAll Job Openings

1 - 10 employees

Founded 2021

🤖 Artificial Intelligence

Artificial Intelligence • Software

Magic is an innovative company focused on building frontier code models to automate software engineering and research. They are working towards achieving safe Artificial General Intelligence (AGI) by automating AI research and code generation, aiming to solve alignment issues more reliably than human efforts alone. The company employs advanced techniques such as frontier-scale pre-training, domain-specific reinforcement learning, and ultra-long context computation. With significant backing, including $515 million in funding, Magic is a small but dedicated team of engineers and researchers addressing fundamental research problems related to AGI.

📋 Description

• Develop new benchmarks and evaluation methodologies • Run data ablations • Assess dataset quality • Investigate whether measurements accurately reflect model performance • Design and validate evaluations and benchmarks across model training and deployment • Investigate model behavior and identify gaps in existing evaluations • Identify opportunities to improve training data • Improve evaluation correctness, reproducibility, and reliability • Audit and improve public benchmarks, research papers, and open-source implementations • Define metrics that accurately reflect model quality • Build tools that make evaluation results accessible and useful across Magic

🎯 Requirements

• Experience developing model evaluations, benchmarks, or ML experiments • Strong programming skills and experience working with machine learning systems • Track record of owning technical projects end-to-end • Skepticism toward results that cannot be reproduced, validated, or explained • Experience analyzing model performance and interpreting experimental results • Comfortable navigating ambiguity and determining whether a measurement actually captures the behavior it claims to measure • Commitment to raising the standard of rigor in AI evaluations

🏖️ Benefits

• Equity is a significant part of total compensation, in addition to salary • 401(k) plan with 6% salary matching • Generous health, dental, and vision insurance for you and your dependents • Unlimited paid time off • Visa sponsorship and relocation support for candidates moving to San Francisco • A small, fast-moving, highly collaborative team working on frontier AI systems

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume