Machine Learning Engineer

Job not on LinkedIn

🔥 11 minutes ago

🏄 California – Remote

🏄 California – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

🤖 Machine Learning Engineer

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Nous Research

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2023

🤖 Artificial Intelligence

🔬 Science

💰 $50M Series A on 2025-05

Artificial Intelligence • Science

Nous Research is a US-based leader in the open-source AI movement that trains large open-source language models and builds infrastructure to coordinate distributed, unbiased training. Its stated mission is to advance human rights and freedoms by creating and proliferating open-source language models, supporting their unrestricted availability and use, and furthering scientific and popular understanding of these models. The company focuses on applied AI research areas including model architecture, data synthesis, fine-tuning, and reasoning, and provides tooling and platforms (e. g. , Hermes, Nous Portal, Psyche, Nous Chat, simulators) to accelerate model development and deployment.

📋 Description

• Run the full eval pipeline end to end and reproduce known results during onboarding, pairing with a senior engineer on your first task • Build a judge calibration protocol: sample human-labeled decisions, measure agreement (κ, per-class P/R), identify drift zones, and document it so anyone can re-run it • Extend an existing benchmark (GAIA, τ-Bench, SWE-bench slice, etc.) with new tasks targeting known capability gaps, including the prompt, environment, rubric, automated grader, and QA • Run failure analysis on model outputs: categorize failure modes, quantify prevalence, and write up findings with recommendations for training data, judge prompts, or benchmark changes • Own a recurring eval workflow (weekly regression suite, judge drift dashboard, red-team evaluation for a new capability) and ship tooling researchers actually use

🎯 Requirements

• 3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job • Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short • Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents • Solid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it • Comfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job) • Understanding of basic eval statistics: why accuracy misleads on imbalanced judges, what Cohen's κ measures, how to think about confidence intervals on a metric • At least 3 of the following: you can explain why LLM-as-judge needs calibration; you've done failure analysis and can tell model bugs apart from prompt, grader, or retrieval issues; you know at least two agent benchmarks (GAIA, AgentBench, τ-Bench, MINT, SWE-bench, WebShop, ALFWorld) and a limitation of each; you've designed or extended an eval dataset with happy paths, edge cases, and adversarial examples; you've thought about non-determinism in eval, how you sample, how many runs, how you report variance • You communicate clearly to both researchers and engineers, in the right language for each • You're comfortable with ambiguity, can turn a half-formed request into a plan, and know when to ask for help.