Senior Product Manager, AI Evaluation – Quality

Job not on LinkedIn

🔥 10 minutes ago

🇺🇸 United States – Remote

💵 $150k - $159k / year

⏰ Full Time

🟠 Senior

✅ Product Manager

👻 Ghost score 0%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Circana

WebsiteLinkedInAll Job Openings

5001 - 10000 employees

🤝 B2B

💼 Consulting

B2B • Consulting

Circana is a market research company that specializes in measuring and accelerating demand for brands. They utilize big data and an AI data-visualization platform to tackle complex business challenges, guiding clients through innovation and meeting consumer demand. With over a century of combined experience from their legacy as IRI and NPD, Circana serves more than 7,000 global brands and retailers, providing insights into both online and in-store purchasing behavior. Their expertise is recognized by prominent media outlets including Bloomberg, CNBC, and The New York Times.

📋 Description

• Define evaluation strategies, methodologies, metrics, and acceptance thresholds for Circana’s AI agents • Build evaluation frameworks for AI agent and agentic workflow capabilities • Partner with the Liquid AI Product Leader, engineering, and product verticals to validate agent performance before release • Develop representative test scenarios, golden datasets, benchmarks, and scoring approaches, including LLM-as-a-judge methods • Evaluate agent quality, accuracy, consistency, reliability, safety, and client value • Identify failure patterns and translate findings into product improvement recommendations • Establish ongoing monitoring of production traces to detect post-launch performance changes • Document evaluation results and communicate findings to technical and business audiences • Promote consistent evaluation standards across Liquid AI product verticals

🎯 Requirements

• 4–6+ years of experience in product evaluation, data science, analytics, experimentation, quality assurance, or a related field • Understanding of statistical evaluation, testing methodologies, and performance metrics • Familiarity with evaluating AI agents or LLM-based systems, including LLM-as-a-judge scoring and production trace analysis • Ability to design structured tests for complex or ambiguous use cases • Strong analytical reasoning and attention to detail • Comfortable operating in an early-stage, fast-moving environment with evolving requirements • Ability to communicate technical findings clearly to varied audiences • Bachelor’s degree in a quantitative, technical, or related discipline, or equivalent experience

🏖️ Benefits

• Flexible and accommodating work environment • Paid time off • Medical, dental, and vision insurance • 401(k) • Bonus pay eligibility • Reasonable accommodation support for applicants with disabilities • Background check conducted in accordance with local legislation • AI-assisted resume screening