Program Manager, Human Evaluations – Recommendations

🕒 October 20

🇺🇸 United States – Remote

💵 $230k - $360k / year

⏰ Full Time

🟡 Mid-level

🟠 Senior

📋 Program Manager

👻 Ghost score 0%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Open for 23 days.
• Jobs at this company usually fill in 50 days.

Apply Now

Find Similar Remote Jobs

Netflix

WebsiteLinkedInAll Job Openings

10,000+ employees

Founded 1997

📱 Media

👥 B2C

Media • B2C

Netflix is a global streaming entertainment company and content producer whose stated mission is "to entertain the world. " It operates a consumer-facing subscription platform offering on-demand TV shows, films, and original programming, and also runs a public careers site emphasizing culture, inclusion, and hiring accommodations. The provided text highlights Netflix’s focus on recruiting talent worldwide, its work-life and culture pages, and its public-facing employer materials.

📋 Description

• Lead end-to-end execution of human evaluation and data operations initiatives from intake and scoping to delivery • Partner with Product Management to translate recommendation strategy and roadmap priorities into evaluation designs, success metrics, and timelines • Manage human annotation vendors, including capacity planning, SLAs, quality performance, and escalations • Drive operational efficiency across evaluation programs, including cost per task, throughput, turnaround time, and speed to IRR • Develop clear, scalable guidelines and scoring rubrics for consistent rater judgment • Oversee rater onboarding, calibration, and QA workflows • Pilot and refine evaluation tasks to improve clarity, inter-rater reliability, and feedback quality • Anticipate and resolve bottlenecks and blockers • Coordinate product, ML, data science, and vendor partners for aligned execution of evaluations at scale • Act as a thought partner to Product Management and help build shared language around recommendation quality

🎯 Requirements

• 4+ years of experience working in human evaluations, data collection, labeling, or annotation operations in ML environments • Track record of implementing process improvements or quality control systems for data collection needs • Prior experience managing human annotation vendors, raters, or data labeling teams, including vendor performance and cost management • Experience improving operational efficiency of annotation programs, including throughput, turnaround, and cost • Strong understanding of evaluation design, including guidelines, rubrics, and scoring protocols • Working fluency in film and TV content, including genres, formats, and talent • Proven ability in end-to-end management of complex, cross-functional programs • Strong Program Management skills and clear accountability for successful delivery • Experience with human labeling platforms • Excellent written and verbal communication skills • Ability to synthesize feedback into clear recommendations and process improvements • Familiarity with rater subjectivity and bias in taste-based judgments and evaluation design • Strong organizational skills and executional focus • Experience with personalization, recommender systems, search, or ranking evaluation is nice to have • Familiarity with offline human evaluation, online experimentation, and engagement metrics is nice to have • Experience evaluating at the list or page level is nice to have • Experience partnering with Product Managers and Data Scientists on metric definition is nice to have

🏖️ Benefits

• Health Plans • Mental Health support • 401(k) Retirement Plan with employer match • Stock Option Program • Disability Programs • Health Savings and Flexible Spending Accounts • Family-forming benefits • Life and Serious Injury Benefits • Paid leave of absence programs • Flexible time off for full-time salaried employees

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume