Natural Language Measurement Specialist

🕒 Julho 24

Candidatar-se

Encontrar Vagas Remotas Similares

Configurar Alerta de Email para Novas Vagas

📊 Verifique sua pontuação de currículo para esta vaga

Melhore suas chances de conseguir uma entrevista verificando sua pontuação de currículo antes de se candidatar.

Enviar Currículo

The College Board

SiteLinkedInTodas as Vagas

1001 - 5000 funcionários

Fundada em 1900

📚 Educação

🤝 Sem Fins Lucrativos

Education • Non-profit

O College Board é uma organização sem fins lucrativos, orientada por sua missão de promover a excelência e a equidade na educação. A instituição conecta estudantes ao sucesso acadêmico e a oportunidades ao apoiá-los na construção de habilidades essenciais para o ensino superior e na remoção de barreiras de acesso. O College Board opera com foco em respeito, diversidade, inclusão e colaboração, contando com uma equipe diversa e dedicada de profissionais. Com sede em várias localidades nos Estados Unidos, a organização busca modernizar suas operações e aumentar sua presença digital, apoiando estudantes e famílias em todo o país.

Descrição

• Help define and advance how language-based performance is measured in high-stakes educational settings • Translate measurement constructs into LLM-based feedback and annotation systems • Design and conduct studies that establish their validity, fairness, and reliability • Serve as the measurement authority for cross-functional partners—psychometricians, engineers, and data scientists • Apply psychometric principles—reliability, validity, dimensionality, and measurement invariance—to LLM-based feedback and annotation systems • Develop and apply methods for detecting and mitigating bias in language-based scores • Establish/recommend annotation guidelines, feedback quality criteria, and standards for acceptable model performance • Contribute to prompt design, fine-tuning, and evaluation of LLM-based feedback and annotation systems • Collaborate with senior team members to translate measurement findings into production systems • Lead and contribute to research studies that evaluate model performance and support assessment validity

🎯 Requisitos

• A Master’s or PhD (or near completion) in a quantitative field such as Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, Data Science, or a related discipline (measurement-focused training strongly valued) • A solid foundation in measurement theory, including reliability, validity, and fairness; familiarity with IRT, generalizability theory, or related frameworks is highly desirable • Experience analyzing language or text data, with exposure to NLP or large language models • Programming skills in Python and familiarity with data science libraries (e.g., pandas, NumPy, PyTorch) • Demonstrated ability to conduct rigorous research (e.g., thesis, publications, or applied research projects) • Strong skills in statistical analysis and experimental design • Familiarity with working with structured and unstructured data • Strong attention to detail and a commitment to producing high-quality, reproducible work • The ability to travel 5-10 times a year to College Board offices or on behalf of College Board business. • Curious and eager to learn new tools, methods, and domains • Thoughtful about the implications of AI systems, including fairness and validity • Able to communicate technical and measurement concepts clearly to diverse audiences • Comfortable working in a collaborative, cross-functional environment

🏖️ Benefícios

• Annual bonuses and opportunities for merit-based raises and promotions • A mission-driven workplace where your impact matters • A team that invests in your development and success