Machine Learning Engineer, Evals

Job not on LinkedIn

🕒 July 28

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Nous Research

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2023

🤖 Artificial Intelligence

🔬 Science

💰 $50M Series A on 2025-05

Artificial Intelligence • Science

Nous Research is a US-based leader in the open-source AI movement that trains large open-source language models and builds infrastructure to coordinate distributed, unbiased training. Its stated mission is to advance human rights and freedoms by creating and proliferating open-source language models, supporting their unrestricted availability and use, and furthering scientific and popular understanding of these models. The company focuses on applied AI research areas including model architecture, data synthesis, fine-tuning, and reasoning, and provides tooling and platforms (e. g. , Hermes, Nous Portal, Psyche, Nous Chat, simulators) to accelerate model development and deployment.

📋 Description

• Run the full eval pipeline end to end and reproduce known results during onboarding, pairing with a senior engineer on the first task • Build a judge calibration protocol using human-labeled decisions, agreement metrics, drift-zone identification, and reproducible documentation • Extend benchmarks such as GAIA, τ-Bench, and SWE-bench with tasks targeting capability gaps, including prompts, environments, rubrics, automated graders, and QA • Analyze model-output failures, categorize failure modes, quantify prevalence, and recommend changes to training data, judge prompts, or benchmarks • Own recurring evaluation workflows such as regression suites, judge-drift dashboards, and red-team evaluations • Ship evaluation tooling used by researchers

🎯 Requirements

• 3+ years in software engineering, ML engineering, data science, or a research-adjacent role • Concrete evaluation experience from coursework, an internship, a side project, open source work, or a job • Experience with at least one LLM evaluation framework, such as Harbor or Nemo Evaluator • Hands-on LLM experience with prompting and few-shot design; ideally fine-tuning or RAG; regular use of coding agents • Solid Python and ability to write clean, tested, version-controlled code • Comfort with Git, CI/CD basics, Docker, and the Linux command line, including SSH, tmux, and debugging remote jobs • Understanding of basic evaluation statistics, including accuracy on imbalanced judges, Cohen's κ, and confidence intervals • At least 3 specified evaluation competencies involving judge calibration, failure analysis, agent benchmarks, eval dataset design, or non-determinism and variance reporting • Clear communication with researchers and engineers • Comfort with ambiguity, planning from half-formed requests, and knowing when to ask for help • Preferred: RLVR/RLHF pipeline experience • Preferred: training data curation experience • Preferred: distributed eval orchestration experience • Preferred: benchmark design from scratch • Preferred: red teaming and adversarial evaluation experience • Preferred: familiarity with psychometrics or measurement theory