Executive Director, Natural Language Measurement

September 24

Apply Now

Receive Emails with Similar Jobs

The College Board

WebsiteLinkedInAll Job Openings

Education • Non-profit

The College Board is a mission-driven, not-for-profit organization dedicated to promoting excellence and equity in education. It connects students to college success and opportunity by supporting them in building essential skills for higher education and removing barriers to access. The College Board operates with a focus on respect, diversity, inclusion, and collaboration, with a diverse and dedicated team of professionals. Based in several locations across the United States, they strive to modernize operations and increase their digital footprint, supporting students and families nationwide.

1001 - 5000 employees

Founded 1900

📚 Education

🤝 Non-profit

📋 Description

• Shape and implement the multi-year strategy and roadmap for automated scoring and intelligent learning approaches across SAT, AP, CLEP, Accuplacer and future products. • Advance College Board’s innovation agenda by expanding from automated scoring to broader AI-enabled solutions that support personalized learning and actionable feedback. • Establish model-readiness criteria and guide release decisions for senior leadership to ensure high-quality, scalable deployment of natural language-driven solutions. • Partner closely with Technology to shape development of guided scoring products and other intelligent learning innovations. • Lead scientific design, implementation, and evaluation of automated scoring and AI/NL applications, ensuring alignment with program and organizational goals. • Apply psychometric and measurement principles to ensure all AI/NL-driven applications uphold validity, fairness, reliability, and equity. • Guide implementation of initial automated scoring programs beginning in 2026 and establish operational processes including monitoring, human-in-the-loop procedures, and evaluation criteria. • Build organizational readiness for adoption of new technologies and establish criteria for model evaluation, quality assurance and responsible release decisions. • Build and lead a team of psychometricians, language model experts, and related specialists; set vision and priorities, track progress, provide coaching, and cultivate an inclusive high-achieving culture. • Serve as College Board’s lead subject-matter expert on AI powered natural language models; represent the organization externally, communicate responsible use guidance, and collaborate across divisions.

🎯 Requirements

• At least 10 years of experience in psychometrics, assessment, or measurement-related fields, with a strong record of applying measurement science in large-scale educational contexts and leading interdisciplinary innovation across psychometrics, AI, and product development. • Deep expertise in measurement science and familiarity with AI/NL approaches, with the ability to apply principles of validity, reliability, fairness, and equity in large-scale assessments. • Proven ability to move AI powered natural language systems from research into operational use, including shaping automated scoring strategies and collaborating with technology and operations partners to ensure quality, fairness, efficiency and integrity. • Demonstrated success setting strategy and guiding innovation through cross-functional collaboration, leading teams across psychometricians, data scientists, engineers, and program staff to execute complex initiatives at scale. • At least 5 years of experience leading and managing direct teams with diverse expertise in measurement, technology, and product, with a strong track record of hiring, developing, and retaining high-performing talent. • Doctorate in psychometrics, educational measurement, statistics, psychology, or a related field. • Exceptional communication and organizational influence skills. • Deep commitment to responsible use of artificial intelligence and to advancing equity in educational measurement. • A strategic and inclusive leadership style and proven ability to drive performance and growth. • The ability to travel 10-12 times a year to College Board offices or on behalf of College Board business. • Authorization to work in the United States for any employer.

🏖️ Benefits

• Annual bonuses and opportunities for merit-based raises and promotions • Competitive compensation and comprehensive package designed to help you thrive • A mission-driven workplace where your impact matters • A team that invests in your development and success • Remote-first workplace policy