Natural Language Measurement Specialist

🕒 July 24

🇺🇸 United States – Remote

💵 $88k - $145k / year

⏰ Full Time

🟡 Mid-level

🟠 Senior

👻 Ghost score 1%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

The College Board

WebsiteLinkedInAll Job Openings

1001 - 5000 employees

Founded 1900

📚 Education

🤝 Non-profit

Education • Non-profit

The College Board is a mission-driven, not-for-profit organization dedicated to promoting excellence and equity in education. It connects students to college success and opportunity by supporting them in building essential skills for higher education and removing barriers to access. The College Board operates with a focus on respect, diversity, inclusion, and collaboration, with a diverse and dedicated team of professionals. Based in several locations across the United States, they strive to modernize operations and increase their digital footprint, supporting students and families nationwide.

📋 Description

• Help define and advance how language-based performance is measured in high-stakes educational settings • Translate measurement constructs into LLM-based feedback and annotation systems • Design and conduct studies that establish their validity, fairness, and reliability • Serve as the measurement authority for cross-functional partners—psychometricians, engineers, and data scientists • Apply psychometric principles—reliability, validity, dimensionality, and measurement invariance—to LLM-based feedback and annotation systems • Develop and apply methods for detecting and mitigating bias in language-based scores • Establish/recommend annotation guidelines, feedback quality criteria, and standards for acceptable model performance • Contribute to prompt design, fine-tuning, and evaluation of LLM-based feedback and annotation systems • Collaborate with senior team members to translate measurement findings into production systems • Lead and contribute to research studies that evaluate model performance and support assessment validity

🎯 Requirements

• A Master’s or PhD (or near completion) in a quantitative field such as Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, Data Science, or a related discipline (measurement-focused training strongly valued) • A solid foundation in measurement theory, including reliability, validity, and fairness; familiarity with IRT, generalizability theory, or related frameworks is highly desirable • Experience analyzing language or text data, with exposure to NLP or large language models • Programming skills in Python and familiarity with data science libraries (e.g., pandas, NumPy, PyTorch) • Demonstrated ability to conduct rigorous research (e.g., thesis, publications, or applied research projects) • Strong skills in statistical analysis and experimental design • Familiarity with working with structured and unstructured data • Strong attention to detail and a commitment to producing high-quality, reproducible work • The ability to travel 5-10 times a year to College Board offices or on behalf of College Board business. • Curious and eager to learn new tools, methods, and domains • Thoughtful about the implications of AI systems, including fairness and validity • Able to communicate technical and measurement concepts clearly to diverse audiences • Comfortable working in a collaborative, cross-functional environment

🏖️ Benefits

• Annual bonuses and opportunities for merit-based raises and promotions • A mission-driven workplace where your impact matters • A team that invests in your development and success