Natural Language Measurement Specialist

🔥 6 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

The College Board

WebsiteLinkedInAll Job Openings

1001 - 5000 employees

Founded 1900

📚 Education

🤝 Non-profit

Education • Non-profit

The College Board is a mission-driven, not-for-profit organization dedicated to promoting excellence and equity in education. It connects students to college success and opportunity by supporting them in building essential skills for higher education and removing barriers to access. The College Board operates with a focus on respect, diversity, inclusion, and collaboration, with a diverse and dedicated team of professionals. Based in several locations across the United States, they strive to modernize operations and increase their digital footprint, supporting students and families nationwide.

📋 Description

• Help define and advance how language-based performance is measured in high-stakes educational settings • Translate measurement constructs into LLM-based feedback and annotation systems • Design and conduct studies that establish their validity, fairness, and reliability • Serve as the measurement authority for cross-functional partners—psychometricians, engineers, and data scientists • Apply psychometric principles—reliability, validity, dimensionality, and measurement invariance—to LLM-based feedback and annotation systems • Develop and apply methods for detecting and mitigating bias in language-based scores • Establish/recommend annotation guidelines, feedback quality criteria, and standards for acceptable model performance • Contribute to prompt design, fine-tuning, and evaluation of LLM-based feedback and annotation systems • Collaborate with senior team members to translate measurement findings into production systems • Lead and contribute to research studies that evaluate model performance and support assessment validity

🎯 Requirements

• A Master’s or PhD (or near completion) in a quantitative field such as Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, Data Science, or a related discipline (measurement-focused training strongly valued) • A solid foundation in measurement theory, including reliability, validity, and fairness; familiarity with IRT, generalizability theory, or related frameworks is highly desirable • Experience analyzing language or text data, with exposure to NLP or large language models • Programming skills in Python and familiarity with data science libraries (e.g., pandas, NumPy, PyTorch) • Demonstrated ability to conduct rigorous research (e.g., thesis, publications, or applied research projects) • Strong skills in statistical analysis and experimental design • Familiarity with working with structured and unstructured data • Strong attention to detail and a commitment to producing high-quality, reproducible work • The ability to travel 5-10 times a year to College Board offices or on behalf of College Board business. • Curious and eager to learn new tools, methods, and domains • Thoughtful about the implications of AI systems, including fairness and validity • Able to communicate technical and measurement concepts clearly to diverse audiences • Comfortable working in a collaborative, cross-functional environment

🏖️ Benefits

• Annual bonuses and opportunities for merit-based raises and promotions • A mission-driven workplace where your impact matters • A team that invests in your development and success