Staff AI Evaluation Lead

🔥 3 hours ago

🇺🇸 United States – Remote

💵 $129k - $152k / year

⏰ Full Time

🟠 Senior

🤖 Artificial Intelligence

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 0%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Fetch Rewards

WebsiteLinkedInAll Job Openings

501 - 1000 employees

Founded 2013

💼 Consulting

🍽️ Food & Beverage

📦 Logistics

💰 Debt Financing on 2022-04

Consulting • Food & Beverage • Logistics

Fetch Rewards is a company that operates America's Rewards App, leveraging the power of Fetch Points to drive consumer behavior and engagement across various industries, including Consumer Packaged Goods (CPG), retail, and restaurants. By offering custom-fit marketing solutions, Fetch helps brands increase sales, improve retention, and build lifelong consumer loyalty without relying on discounts. The platform offers dynamic in-app offers based on verified shopping behavior, designed to inspire specific business outcomes and increase customer engagement. Fetch Rewards partners with major brands and retailers, providing a robust data-driven approach to enhance buying frequency and customer retention.

📋 Description

• Lead complex, high-impact automation and evaluation initiatives across workflows or teams • Architect end-to-end workflows integrating datasets, evaluations, automations, and human-in-the-loop processes • Build reusable harnesses, components, and evaluation pipelines that run against production without hands-on operation • Define dataset standards, evaluation methodology, failure taxonomy, and quality measurement across AI Operations • Verify that projects meet established standards and own the standards creation and adoption process • Create, maintain, validate, and revise organizational metric definitions • Define production quality thresholds and determine where human review remains in the loop • Allocate evaluation depth by system risk and make accepted tradeoffs visible to stakeholders • Define re-baselining requirements for model and platform changes • Evaluate new models and AI capabilities and decide what the organization adopts • Lead redesign of workflows, tooling, and processes to improve performance and durability across AI Operations • Partner with Engineering, Product, and AI teams to deliver solutions and align priorities • Define success metrics and communicate performance and recommendations to leadership • Mentor others and strengthen team automation and evaluation capabilities • Stay current on emerging tools and approaches, pilot them, and translate them into improvements

🎯 Requirements

• 8+ years of professional experience in AI, machine learning, operational automations, or a related field • Proven ability to lead complex automation or evaluation initiatives across systems or teams • Experience designing, building, and scaling evaluation frameworks, datasets, and quality systems at scale for LLMs or AI products • Fluency in SQL, JSON, APIs, and scripting with AI assistance • Ownership of the technical direction of the evaluation stack • Deep working knowledge of LLM and agentic system behavior, automation platforms, and system design, demonstrated in systems built • Experience with data pipelines, APIs, and production systems • Experience influencing cross-functional stakeholders and aligning priorities • Demonstrated ability to define metrics and drive measurable business impact • Experience mentoring or leading technical contributors (preferred) • Experience evaluating agentic systems in production at scale (preferred) • Experience setting technical standards adopted across an organization (preferred)

🏖️ Benefits

• Equity for full-time employees • 401(k) matching dollar for dollar, up to 4% of eligible pay • Comprehensive medical, dental, and vision plans, including pets • $10,000 per year in education reimbursement • Employee Resource Groups and Inclusion Council participation • Flexible PTO • 9 paid holidays • Year-end week-long break • 14 weeks of fully paid parental leave • 6 additional weeks of paid childbirth recovery leave for birthing parents • Flexible return-to-work schedule • One-time $2,000 Calvin Care Cash incentive for employees welcoming new family members • Flexible work environment • Hardware and software provided for home working

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume