Applied Research Scientist, LLM Evaluation – Post-Training

Job not on LinkedIn

🔥 3 hours ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

InnoData

WebsiteLinkedInAll Job Openings

2 - 10 employees

Founded 2019

🤝 B2B

💼 Consulting

🌍 Social Impact

B2B • Consulting • Social Impact

InnoData is INNOvation DATA SCS, an Italian social cooperative based in Foggia that identifies itself as a provider of technological solutions. The company website (currently under maintenance) highlights "Soluzioni tecnologiche" (technological solutions) and emphasizes social impact ("Impatto sociale"). Contact details listed include Via Francesco Crispi 65, 71121 Foggia, Italy. Based on the available information, InnoData appears to operate at the intersection of technology and social impact, likely offering tech-focused services to other organizations.

📋 Description

• Lead research and experimentation on how evaluation design, measurement strategies, and feedback signals influence model improvement. • Define and execute a research agenda focused on LLM evaluation and post-training, especially evaluation-driven model improvement. • Design rigorous experiments to study how evaluation methodologies impact fine-tuning and post-training outcomes. • Develop and validate evaluation frameworks for LLM and multimodal systems, including benchmark/task design, scoring methods, judge/model-assisted evaluation, and human evaluation protocols. • Analyze model behavior and failure patterns; generate actionable recommendations for model improvement and evaluation redesign. • Collaborate with AI/ML Research Engineers to translate research methods into scalable evaluation and post-training pipelines. • Engage with customer technical stakeholders to understand evaluation goals, review methodologies, and provide expert recommendations. • Produce high-quality technical documentation, internal research reports, and client-facing materials explaining methods, results, assumptions, and limitations.

🎯 Requirements

• MS/PhD in Computer Science, Machine Learning, Statistics, Applied Mathematics, AI, or a related quantitative scientific field (PhD strongly preferred) • 5+ years of relevant experience in applied research / research science in ML/AI, with substantial work in LLMs or foundation models • Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality research • Strong foundation in experimental design, statistical analysis, and scientific reasoning for ML systems • Strong coding skills in Python for research experimentation and analysis (e.g., data processing, evaluation pipelines, statistical analysis, visualization) • Experience working with modern ML tooling/frameworks (e.g., PyTorch, Hugging Face, JAX/TensorFlow as applicable) sufficient to design and execute model/evaluation experiments • Ability to evaluate and compare human and automated evaluation methods, including tradeoffs in cost, reliability, validity, and scalability • Experience designing evaluation studies and protocols that are reproducible across datasets, model versions, and evaluation runs • Ability to collaborate directly with technical stakeholders including research scientists, ML engineers, data scientists, and customer technical counterparts • Strong communication skills and ability to present nuanced technical conclusions, assumptions, and limitations clearly.