Staff AI Evaluation Lead

🔥 47 minutes ago

🇺🇸 United States – Remote

💵 $129k - $152k / year

⏰ Full Time

🟠 Senior

🤖 Artificial Intelligence

👻 Ghost score 0%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Fetch

WebsiteLinkedInAll Job Openings

501 - 1000 employees

Founded 2013

👥 B2C

📣 Marketing

🔥 Funding within the last year

💰 Later Stage VC on 2026-05

B2C • Marketing

Fetch is a consumer mobile rewards app that lets users scan receipts to earn points that can be redeemed for gift cards and other rewards. The app awards points for everyday purchases (groceries, fuel, meals, games, etc. ), offers in-app games and promotions, and provides a business-facing offering (Fetch for Business) and browser extension. Fetch reports millions of monthly earners, millions of app reviews, and over $1B in rewards redeemed. The service emphasizes easy receipt scanning, point accumulation, and low-effort rewards for shoppers.

📋 Description

• Lead complex, high-impact automation and evaluation initiatives across workflows or teams • Architect end-to-end workflows integrating datasets, evaluations, automations, and human-in-the-loop processes • Build reusable harnesses, components, and evaluation pipelines that run against production without hands-on operation • Define dataset standards, evaluation methodology, failure taxonomy, and quality measurement across AI Operations • Verify that projects meet established standards and own the standards creation and adoption process • Create, maintain, validate, and revise organizational metric definitions • Define production quality thresholds and determine where human review remains in the loop • Allocate evaluation depth by system risk and make accepted tradeoffs visible to stakeholders • Define re-baselining requirements for model and platform changes • Evaluate new models and AI capabilities and decide what the organization adopts • Lead redesign of workflows, tooling, and processes to improve performance and durability across AI Operations • Partner with Engineering, Product, and AI teams to deliver solutions and align priorities • Define success metrics and communicate performance and recommendations to leadership • Mentor others and strengthen team automation and evaluation capabilities • Stay current on emerging tools and approaches, pilot them, and translate them into improvements

🎯 Requirements

• 8+ years of professional experience in AI, machine learning, operational automations, or a related field • Proven ability to lead complex automation or evaluation initiatives across systems or teams • Experience designing, building, and scaling evaluation frameworks, datasets, and quality systems at scale for LLMs or AI products • Fluency in SQL, JSON, APIs, and scripting with AI assistance • Ownership of the technical direction of the evaluation stack • Deep working knowledge of LLM and agentic system behavior, automation platforms, and system design, demonstrated in systems built • Experience with data pipelines, APIs, and production systems • Experience influencing cross-functional stakeholders and aligning priorities • Demonstrated ability to define metrics and drive measurable business impact • Experience mentoring or leading technical contributors (preferred) • Experience evaluating agentic systems in production at scale (preferred) • Experience setting technical standards adopted across an organization (preferred)

🏖️ Benefits

• Equity for full-time employees • 401(k) matching dollar for dollar, up to 4% of eligible pay • Comprehensive medical, dental, and vision plans, including pets • $10,000 per year in education reimbursement • Employee Resource Groups and Inclusion Council participation • Flexible PTO • 9 paid holidays • Year-end week-long break • 14 weeks of fully paid parental leave • 6 additional weeks of paid childbirth recovery leave for birthing parents • Flexible return-to-work schedule • One-time $2,000 Calvin Care Cash incentive for employees welcoming new family members • Flexible work environment • Hardware and software provided for home working

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume