Principal Research Scientist

🔥 0 minutes ago

🇺🇸 United States – Remote

💵 $190k - $235k / year

⏰ Full Time

🔴 Lead

🧬 Research Scientist

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 0%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

G2

WebsiteLinkedInAll Job Openings

501 - 1000 employees

Founded 2012

💼 Consulting

📣 Marketing

📦 Logistics

Consulting • Marketing • Logistics

G2 is a business software review platform that provides insights to potential buyers through over 2. 5 million real user reviews. The platform covers a wide range of software categories including project management, video conferencing, e-commerce platforms, CRM, AI chatbots, and many more. G2 empowers businesses to discover and compare software options to make informed purchasing decisions. It serves as a resource for marketers, sales professionals, IT and finance departments to optimize their software investments. Additionally, G2 offers solutions for software vendors to enhance their profiles and reach more buyers.

📋 Description

• Own the evaluation methodology and set standards for evaluating AI agents • Work with engineers and product managers to build and improve evaluations in G2’s current offerings • Scope features, turn methods into production-ready components, and iterate based on customer and business needs • Design realistic environments and task sets reflecting business work • Develop accurate, reproducible and difficult-to-game scoring approaches using automated checks, LLM-based judges and human review • Measure consistency and reliability of evaluations • Audit tasks and graders, quantify noise, and establish when agent differences are real • Present and defend evaluation findings to buyers, customers, partners and the research community • Mentor and guide researchers and engineers on evaluation design and measurement • Review work and document standards for repeatable evaluations • Set and pursue a research agenda across agent evaluation, user simulation, LLM-as-judge reliability, voice and multimodal agents, long-horizon workflows and test-set contamination • Follow new benchmarks, papers and evaluation practices and translate them into product and roadmap decisions • Write methodology documentation, papers and public posts where appropriate • Partner with data and go-to-market teams on presentation and use of results

🎯 Requirements

• Experience working with evaluation design or dataset curation in multi-agentic systems • 5+ years building or evaluating ML systems, including hands-on work with LLMs or LLM-based agents • 2+ years experience managing small to medium teams preferred • A graduate degree (MS or PhD) in machine learning, NLP, computer science, statistics or a related field, or equivalent applied research experience • Track record of original ideas in machine learning, shown through first-author publications or projects led end-to-end • Ability to own a research agenda and carry long-running projects independently • Track record of shipping ML work in a product setting with engineers and product managers • Experience setting technical direction, mentoring researchers or engineers, and reviewing methodology • Familiarity with AI benchmarking and benchmark limitations • Sound statistical judgment, including experimental design, variance, significance and agreement between human and automated raters • Good judgment about practical trade-offs • Clear writing and ability to explain methodology to non-specialists and defend it to skeptical audiences • Experience in agent evaluation, user simulation, or human and AI-based judging is helpful • Experience with conversational, voice, retrieval-based or tool-using AI systems is helpful • Contributions to open-source evaluation tools or public leaderboards are helpful • Familiarity with business software workflows is helpful • Publications at ACL, EMNLP, NeurIPS, ICML, ICLR, COLM or similar venues are helpful

🏖️ Benefits

• Equity • Flexible work • Ample parental leave • Unlimited PTO