Director, AI Evaluation

Job not on LinkedIn

🔥 0 minutes ago

🔔 Pennsylvania – Remote

🔔 Pennsylvania – Remote

⏰ Full Time

🔴 Lead

🤖 Artificial Intelligence

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Geisinger

WebsiteLinkedInAll Job Openings

10,000+ employees

Founded 1915

💊 Pharmaceuticals

🧘 Wellness

Healthcare • Pharmaceuticals • Wellness

Geisinger is a healthcare organization that has been providing accessible medical services for over a century in Pennsylvania. It focuses on meeting the healthcare needs of its communities and is dedicated to innovative patient care. With career opportunities in various fields including nursing, allied health, and administration, Geisinger promotes professional development and a supportive workplace for its employees, emphasizing diversity, equity, and inclusion.

📋 Description

• The Director of AI Evaluation owns how Geisinger defines, proves, and sustains quality across its entire AI portfolio — internally built models and vendor-provided systems alike. • Manages the people who do the building and the proving: the data scientists who develop production machine learning and fine-tuned AI systems, and the senior analysts who evaluate them. • Sets that standard, leads the team that enforces it, and reports findings to the VP of AI, executive leaders, and the board. • Owns the methodology that holds initiatives to that standard: pre-production validation and live production monitoring. • Attracts and retains strong technical talent, developing careers, and leads the evaluation team. • Provides hands-on technical guidance to program teams as they design validation studies, equity audits, monitoring plans, and escalation playbooks. • Owns the evaluation toolkit and reusable playbooks and templates that let each new program move faster than the last. • Connects each AI to the Outcome it was deployed to improve against a pre-launch baseline.

🎯 Requirements

• People-leadership experience — managing, developing, and growing technical staff; building teams, not just leading projects. • Strong foundation in experimental design and causal inference, with judgment about which method fits which situation. • Hands-on experience designing and running model evaluation studies in real production settings. • Experience evaluating LLM or generative AI systems, or comparable experience with complex ML systems where ground truth is ambiguous or noisy. • Proven ability to translate ambiguous failure modes into concrete, defensible evaluation designs and monitoring metrics. • Strong fluency in Python and SQL; working comfort with modern ML tooling and cloud-native data environments. • Experience in evaluating fairness and equity in ML systems. • Clear written communication — the role produces evaluation memos and specifications that non-technical decision-makers rely on. • Healthcare, clinical, or regulated-industry experience strongly preferred.

🏖️ Benefits

• We offer healthcare benefits for full time and part time positions from day one, including vision, dental and domestic partners. • We encourage an atmosphere of collaboration, cooperation and collegiality.