Staff AI Evaluation Lead

πŸ•’ October 30

Where this job comes from

Posted on Fetch Rewards's careers page β†—Fetch Rewards's careers page, not through a job board

First seen by Remote Rocketship October 30

Still open on Fetch Rewards's site. Checked 1 hour ago.

Apply Now
Logo of Fetch Rewards

Fetch Rewards

501 - 1000 employees

Founded 2013

πŸ’Ό Consulting

🍽️ Food & Beverage

πŸ“¦ Logistics

πŸ’° Debt Financing on 2022-04

Consulting β€’ Food & Beverage β€’ Logistics

Fetch Rewards is a company that operates America's Rewards App, leveraging the power of Fetch Points to drive consumer behavior and engagement across various industries, including Consumer Packaged Goods (CPG), retail, and restaurants. By offering custom-fit marketing solutions, Fetch helps brands increase sales, improve retention, and build lifelong consumer loyalty without relying on discounts. The platform offers dynamic in-app offers based on verified shopping behavior, designed to inspire specific business outcomes and increase customer engagement. Fetch Rewards partners with major brands and retailers, providing a robust data-driven approach to enhance buying frequency and customer retention.

πŸ“‹ Description

β€’ Lead complex, high-impact automation and evaluation initiatives across workflows or teams β€’ Architect end-to-end workflows integrating datasets, evaluations, automations, and human-in-the-loop processes β€’ Build reusable harnesses, components, and evaluation pipelines that run against production without hands-on operation β€’ Define dataset standards, evaluation methodology, failure taxonomy, and quality measurement across AI Operations β€’ Verify that projects meet established standards and own the standards creation and adoption process β€’ Create, maintain, validate, and revise organizational metric definitions β€’ Define production quality thresholds and determine where human review remains in the loop β€’ Allocate evaluation depth by system risk and make accepted tradeoffs visible to stakeholders β€’ Define re-baselining requirements for model and platform changes β€’ Evaluate new models and AI capabilities and decide what the organization adopts β€’ Lead redesign of workflows, tooling, and processes to improve performance and durability across AI Operations β€’ Partner with Engineering, Product, and AI teams to deliver solutions and align priorities β€’ Define success metrics and communicate performance and recommendations to leadership β€’ Mentor others and strengthen team automation and evaluation capabilities β€’ Stay current on emerging tools and approaches, pilot them, and translate them into improvements

🎯 Requirements

β€’ 8+ years of professional experience in AI, machine learning, operational automations, or a related field β€’ Proven ability to lead complex automation or evaluation initiatives across systems or teams β€’ Experience designing, building, and scaling evaluation frameworks, datasets, and quality systems at scale for LLMs or AI products β€’ Fluency in SQL, JSON, APIs, and scripting with AI assistance β€’ Ownership of the technical direction of the evaluation stack β€’ Deep working knowledge of LLM and agentic system behavior, automation platforms, and system design, demonstrated in systems built β€’ Experience with data pipelines, APIs, and production systems β€’ Experience influencing cross-functional stakeholders and aligning priorities β€’ Demonstrated ability to define metrics and drive measurable business impact β€’ Experience mentoring or leading technical contributors (preferred) β€’ Experience evaluating agentic systems in production at scale (preferred) β€’ Experience setting technical standards adopted across an organization (preferred)

πŸ–οΈ Benefits

β€’ Equity for full-time employees β€’ 401(k) matching dollar for dollar, up to 4% of eligible pay β€’ Comprehensive medical, dental, and vision plans, including pets β€’ $10,000 per year in education reimbursement β€’ Employee Resource Groups and Inclusion Council participation β€’ Flexible PTO β€’ 9 paid holidays β€’ Year-end week-long break β€’ 14 weeks of fully paid parental leave β€’ 6 additional weeks of paid childbirth recovery leave for birthing parents β€’ Flexible return-to-work schedule β€’ One-time $2,000 Calvin Care Cash incentive for employees welcoming new family members β€’ Flexible work environment β€’ Hardware and software provided for home working

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ October 30

Function Health

1001 - 5000

πŸ₯ Healthcare

πŸ‘₯ B2C

Senior Clinical Specialist ensuring Function Health’s AI-generated health guidance is clinically accurate, useful, and safe. Partnering with product and engineering teams to improve AI evaluations and member experiences.

πŸ‡ΊπŸ‡Έ United States – Remote

πŸ”₯ Funding within the last year

πŸ’° $450M Debt Financing on 2026-07

⏰ Full Time

🟠 Senior

πŸ€– Artificial Intelligence

πŸ•’ October 30

Clarinet

11 - 50

πŸ’Ό Consulting

🀝 B2B

πŸ“š Education

Technical AI Learning Facilitator delivering adaptive workshops for Clarinet, an AI enablement consultancy. Building advanced AI fluency, supporting clients, and shaping rapidly evolving learning content.

πŸ•’ October 30

Clarinet

11 - 50

πŸ’Ό Consulting

🀝 B2B

πŸ“š Education

AI Learning Facilitator delivering adaptive workshops on AI tools, adoption, and behavioral change. Supporting Clarinet’s AI enablement consulting, client relationships, content, and thought leadership.

πŸ•’ October 30

Tallwave

51 - 200

πŸ’Ό Consulting

πŸ“£ Marketing

AI Systems Configurator building connected AI workflows for Tallwave, a growth partner helping organizations improve operations and business growth. Automating reporting, client intelligence, and delivery systems.

πŸ•’ October 30

Parallel Works

11 - 50

πŸ€– Artificial Intelligence

☁️ SaaS

πŸ›οΈ Government

AI Compute Sales Lead growing Parallel Works’ managed GPU business for research institutions, laboratories, and AI companies. Building pipeline, selling GPU capacity, and managing technical enterprise deals.

Built byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok