Member of Technical Staff, ML Platform – Evals

🕒 September 22

🇺🇸 United States – Remote

💵 $240k - $290k / year

⏰ Full Time

🔴 Lead

🖥 Software Engineer

👻 Ghost score 26%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Open for 51 days.
• Jobs at this company usually fill in 34 days.
• Reposted 2 times.

Apply Now

Find Similar Remote Jobs

Runway

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2018

🤖 Artificial Intelligence

☁️ SaaS

📱 Media

Artificial Intelligence • SaaS • Media

Runway is a company building foundational "Real-World Intelligence" — large models and systems that can understand, simulate and act in the visual world. It offers three main product platforms built on these models: Runway Creative, an all‑in‑one cloud creative suite for generating and editing video, images and audio for individuals and teams; Runway Dev, an AI media platform and API for developers to run models, compose pipelines and trigger workflows; and Runway Robotics, a toolkit for photorealistic simulation, policy inference and synthetic data generation powered by their General World Model (GWM-1). Runway also publishes research (e. g. , GWM-1, Gen-4. 5 video model) and serves both individual creators and enterprise customers.

📋 Description

• Own the ML evaluation platform end to end, including tooling and systems to generate, annotate, review, and adapt evaluations at frontier scale • Define CLIs, APIs, GUIs, and storage layers for a seamless, fast, sophisticated, and collaborative evaluation process • Work directly with research teams on video, image, audio, agents, and robotics to understand measurement needs and build generalized platform capabilities • Set standards for model evaluation, including reproducibility, metric definitions, reporting formats, and thresholds for trustworthy results • Support broad adoption and integration of the platform across training, production model serving, and other research efforts • Contribute to ML Platform tools and systems that help Runway train and serve frontier models • Generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and present findings to researchers quickly • Define operational measures of model quality and support ship/no-ship decisions

🎯 Requirements

• 5+ years of experience building ML infrastructure or data platforms in production environments • Experience with evaluation, experimentation, or benchmarking systems • Strong Python and PyTorch skills • Hands-on experience running large batch GPU workloads on Kubernetes • Experience designing data pipelines and storage for large volumes of media or model outputs • Familiarity with versioning and reproducibility • Familiarity with experimental statistics, including paired comparisons, confidence intervals, multiple-comparison pitfalls, and inter-rater agreement • Comfort building internal tools end to end, from command line to browser • Ability to lead a broad technical area, gather requirements, set direction, make tradeoffs, and drive a roadmap • Familiarity with the full model development lifecycle: data, training, evaluation, and serving • Ability to work as a self-starter embedded with research teams • Strong systems thinking and pragmatic approach to production reliability • Humility and open mindedness • If located in the US, current authorization to work in the US is requested in the application form • Track record of building evaluation suites for generative models is nice to have • Hands-on work with LLM- or VLM-as-judge pipelines is nice to have • Experience with online experimentation platforms and connecting offline metrics to product outcomes is nice to have • Prior work evaluating agents or robotics policies is nice to have

🏖️ Benefits

• Salary range of $240K–$290K • Access to best-in-class models, agents, GPUs, storage and cloud services • Diverse, inclusive workplace with equal opportunity to succeed • Global research and engineering environment

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume