Senior AI Product Manager – Evals & Annotations

Job not on LinkedIn

🔥 10 minutes ago

🏄 California, New York – Remote

🏄 California – Remote 🗽 New York – Remote

💵 $200k - $250k / year

⏰ Full Time

🟠 Senior

✅ Product Manager

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 11%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Just posted.
• Jobs at this company usually fill in 50 days.

Apply Now

Find Similar Remote Jobs

Arize AI

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2019

🤖 Artificial Intelligence

☁️ SaaS

🏢 Enterprise

Artificial Intelligence • SaaS • Enterprise

Arize AI is a company that provides a comprehensive AI observability and evaluation platform designed for AI engineers to build and manage AI applications, particularly those involving large language models (LLMs). The platform offers end-to-end AI tracing, monitoring, and evaluation tools to help users identify and resolve performance bottlenecks, ensure data quality, and improve the performance of AI systems. Arize also supports experimentation and visualization, enabling users to iterate rapidly on AI applications and maintain high-quality service delivery. The company caters to top AI companies, offering solutions for both online and offline LLM evaluation and observability. Arize's tools are particularly valuable for AI model monitoring, troubleshooting, and performance optimization, helping businesses mitigate risks and improve AI outcomes.

📋 Description

• Own Arize's Evals & Annotations product area end-to-end, including eval creation and execution, human review, annotation workflows, and the broader evaluation lifecycle • Set product strategy and roadmap, balancing customer needs, platform investments, UX improvements, and emerging AI and agent-development opportunities • Develop strong workflow opinions, independently prototype solutions, and partner with Design on intuitive, polished experiences • Shape the future of evaluations for autonomous and managed agents, including opportunities within the broader Dynatrace ecosystem • Work with Engineering on evaluation execution, backend systems, orchestration, and data infrastructure • Contribute to technical discussions, implementation approaches, and tradeoffs • Have multiple customer conversations each week to understand evaluation practices, uncover problems, and validate product direction • Collaborate with cofounders and cross-functional teams on product direction, design exploration, and prioritization • Make prioritization decisions and drive products from problem definition through launch, adoption, and iteration

🎯 Requirements

• Significant product management experience independently owning technically complex developer-facing products; exact years flexible for candidates with exceptional technical depth, product judgment, and ownership • Strong understanding of LLM evaluations and AI development workflows, including automated evals, human feedback, experimentation, and iteration on AI applications and agents • Experience with developer tools, AI infrastructure, ML platforms, or adjacent technical products • Solid understanding of backend systems and software architecture • Strong technical depth and hands-on engineering ability • Ability to debug issues independently and navigate code when needed • Ability to put up small fixes or PRs • Ability to build prototypes and technical proofs of concept • Ability to propose technical implementations and reason through tradeoffs • Ability to hold credible technical discussions with engineers • Excellent product and UX instincts; ability to define and prototype core product experiences • Strong ownership, initiative, and bias toward action • Ability to navigate ambiguity and make tradeoffs • Collaborative, low-ego approach to working with founders, engineers, and designers • Experience working closely with technical customers • Strong communication skills and ability to move between strategy and execution

🏖️ Benefits

• Competitive equity package • Medical insurance • Dental insurance • Vision insurance • 401(k) plan • Unlimited paid time off • Generous parental leave plan • Mental and wellness support • WFH monthly stipend for co-working spaces • Optional in-person work in New York City and the San Francisco Bay Area • Regular chats with industry experts, researchers, and ethicists • Culturally conscious events such as LGBTQ trivia during pride month • Active Lady Arizers subgroup

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume