Member of Technical Staff, Evals

πŸ•’ November 11

πŸ‡ΊπŸ‡Έ United States – Remote

πŸ’΅ $200k - $550k / year

⏰ Full Time

🟑 Mid-level

🟠 Senior

πŸ–₯ Software Engineer

Where this job comes from

Posted on Magic's careers page β†—Magic's careers page, not through a job board

First seen by Remote Rocketship November 11

Still open on Magic's site. Checked 2 hours ago.

Apply Now
Logo of Magic

Magic

1 - 10 employees

Founded 2021

πŸ€– Artificial Intelligence

Artificial Intelligence β€’ Software

Magic is an innovative company focused on building frontier code models to automate software engineering and research. They are working towards achieving safe Artificial General Intelligence (AGI) by automating AI research and code generation, aiming to solve alignment issues more reliably than human efforts alone. The company employs advanced techniques such as frontier-scale pre-training, domain-specific reinforcement learning, and ultra-long context computation. With significant backing, including $515 million in funding, Magic is a small but dedicated team of engineers and researchers addressing fundamental research problems related to AGI.

πŸ“‹ Description

β€’ Develop new benchmarks and evaluation methodologies β€’ Run data ablations β€’ Assess dataset quality β€’ Investigate whether measurements accurately reflect model performance β€’ Design and validate evaluations and benchmarks across model training and deployment β€’ Investigate model behavior and identify gaps in existing evaluations β€’ Identify opportunities to improve training data β€’ Improve evaluation correctness, reproducibility, and reliability β€’ Audit and improve public benchmarks, research papers, and open-source implementations β€’ Define metrics that accurately reflect model quality β€’ Build tools that make evaluation results accessible and useful across Magic

🎯 Requirements

β€’ Experience developing model evaluations, benchmarks, or ML experiments β€’ Strong programming skills and experience working with machine learning systems β€’ Track record of owning technical projects end-to-end β€’ Skepticism toward results that cannot be reproduced, validated, or explained β€’ Experience analyzing model performance and interpreting experimental results β€’ Comfortable navigating ambiguity and determining whether a measurement actually captures the behavior it claims to measure β€’ Commitment to raising the standard of rigor in AI evaluations

πŸ–οΈ Benefits

β€’ Equity is a significant part of total compensation, in addition to salary β€’ 401(k) plan with 6% salary matching β€’ Generous health, dental, and vision insurance for you and your dependents β€’ Unlimited paid time off β€’ Visa sponsorship and relocation support for candidates moving to San Francisco β€’ A small, fast-moving, highly collaborative team working on frontier AI systems

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ November 11

Bright Vision Technologies

51 - 200

πŸ’Ό Consulting

πŸ₯ Healthcare

πŸ”’ Cybersecurity

Oracle PL/SQL Developer building and optimizing enterprise database platforms for a technology consulting company. Supporting cloud modernization, financial, healthcare, and regulatory systems.

πŸ•’ November 11

Zayo Group

1001 - 5000

πŸ“¦ Logistics

πŸ’Ό Consulting

πŸ₯ Healthcare

Managing fiber engineers and planners designing, implementing, and maintaining Zayo’s telecommunications network. Overseeing routes, databases, projects, KPIs, and team development.

πŸ•’ November 11

IRIS Software Group

1001 - 5000

☁️ SaaS

🀝 B2B

πŸ‘₯ HR Tech

Software Engineer building scalable .NET cloud applications for IRIS, a UK software company serving businesses, schools and accountancy firms. Owning features from design through production.

πŸ‡ΊπŸ‡Έ United States – Remote

πŸ’° Private Equity Round on 2018-05

⏰ Full Time

🟒 Junior

🟑 Mid-level

πŸ–₯ Software Engineer

πŸš«πŸ‘¨β€πŸŽ“ No degree required

πŸ•’ November 11

CoLogix Analytics

1 - 10

πŸ₯ Healthcare

πŸ’Ό Consulting

πŸ“¦ Logistics

NetSuite Developer building customizations and integrations for Cologix, a North American interconnection and hyperscale edge data center company. Supporting scalable internal corporate systems.

πŸ•’ November 11

turbopuffer

1 - 10

☁️ SaaS

πŸ”Œ API

πŸ€– Artificial Intelligence

Head of Deployed Engineering leading turbopuffer’s customer-facing technical team. Driving technical sales, demos, benchmarks, implementations, and customer success for database solutions.

Developed byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok