Principal Research Scientist

🕒 October 27

🇺🇸 United States – Remote

💵 $190k - $235k / year

⏰ Full Time

🟡 Mid-level

🟠 Senior

🧬 Research Scientist

Where this job comes from

Posted on G2's careers page ↗G2's careers page, not through a job board

First seen by Remote Rocketship October 27

Still open on G2's site. Checked 4 hours ago.

Apply Now
Logo of G2

G2

501 - 1000 employees

Founded 2012

💼 Consulting

📣 Marketing

📦 Logistics

Consulting • Marketing • Logistics

G2 is a business software review platform that provides insights to potential buyers through over 2. 5 million real user reviews. The platform covers a wide range of software categories including project management, video conferencing, e-commerce platforms, CRM, AI chatbots, and many more. G2 empowers businesses to discover and compare software options to make informed purchasing decisions. It serves as a resource for marketers, sales professionals, IT and finance departments to optimize their software investments. Additionally, G2 offers solutions for software vendors to enhance their profiles and reach more buyers.

📋 Description

• Own the evaluation methodology and set standards for evaluating AI agents • Work with engineers and product managers to build and improve evaluations in G2’s current offerings • Scope features, turn methods into production-ready components, and iterate based on customer and business needs • Design realistic environments and task sets reflecting business work • Develop accurate, reproducible and difficult-to-game scoring approaches using automated checks, LLM-based judges and human review • Measure consistency and reliability of evaluations • Audit tasks and graders, quantify noise, and establish when agent differences are real • Present and defend evaluation findings to buyers, customers, partners and the research community • Mentor and guide researchers and engineers on evaluation design and measurement • Review work and document standards for repeatable evaluations • Set and pursue a research agenda across agent evaluation, user simulation, LLM-as-judge reliability, voice and multimodal agents, long-horizon workflows and test-set contamination • Follow new benchmarks, papers and evaluation practices and translate them into product and roadmap decisions • Write methodology documentation, papers and public posts where appropriate • Partner with data and go-to-market teams on presentation and use of results

🎯 Requirements

• Experience working with evaluation design or dataset curation in multi-agentic systems • 5+ years building or evaluating ML systems, including hands-on work with LLMs or LLM-based agents • 2+ years experience managing small to medium teams preferred • A graduate degree (MS or PhD) in machine learning, NLP, computer science, statistics or a related field, or equivalent applied research experience • Track record of original ideas in machine learning, shown through first-author publications or projects led end-to-end • Ability to own a research agenda and carry long-running projects independently • Track record of shipping ML work in a product setting with engineers and product managers • Experience setting technical direction, mentoring researchers or engineers, and reviewing methodology • Familiarity with AI benchmarking and benchmark limitations • Sound statistical judgment, including experimental design, variance, significance and agreement between human and automated raters • Good judgment about practical trade-offs • Clear writing and ability to explain methodology to non-specialists and defend it to skeptical audiences • Experience in agent evaluation, user simulation, or human and AI-based judging is helpful • Experience with conversational, voice, retrieval-based or tool-using AI systems is helpful • Contributions to open-source evaluation tools or public leaderboards are helpful • Familiarity with business software workflows is helpful • Publications at ACL, EMNLP, NeurIPS, ICML, ICLR, COLM or similar venues are helpful

🏖️ Benefits

• Equity • Flexible work • Ample parental leave • Unlimited PTO

Apply Now

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

🕒 October 26

SandboxAQ

51 - 200

🤖 Artificial Intelligence

🔒 Cybersecurity

💊 Pharmaceuticals

Principal Scientist leading AI-driven discovery of rare-earth-lean permanent magnets. Guiding simulation, experimental validation, IP, and U.S. manufacturing partnerships.

🇺🇸 United States – Remote

💵 $224k - $420k / year

⏰ Full Time

🟠 Senior

🔴 Lead

🧬 Research Scientist

🕒 October 23

GondolaBio

11 - 50

🏥 Healthcare

💼 Consulting

🏭 Manufacturing

Translational medicine scientist advancing biomarker strategies for GondolaBio’s genetic-disease therapeutics. Leading assay validation and clinical implementation across early neuroscience programs.

🇺🇸 United States – Remote

💵 $190k - $210k / year

💰 $300M Venture Round - GondolaBio on 2024-08

⏰ Full Time

🟠 Senior

🧬 Research Scientist

🕒 October 20

Zillow

5001 - 10000

🏠 Real Estate

🛍️ eCommerce

👥 B2C

Applied Research Scientist building Zillow’s AI World Model for housing-market demand and planning. Developing uncertainty-aware models that power next-generation home-shopping experiences.

🇺🇸 United States – Remote

💵 $185.8k - $312.4k / year

💰 $4.1M Post-IPO Equity on 2012-12

⏰ Full Time

🟠 Senior

🧬 Research Scientist

🕒 October 20

Pindrop

201 - 500

🛡️ Insurance

💼 Consulting

🏥 Healthcare

Research Scientist advancing AI-driven voice, fraud, identity, and deepfake security at Pindrop. Developing deep learning models and translating research into production-ready enterprise solutions.

🕒 October 20

Troveo AI

11 - 50

🤖 Artificial Intelligence

📱 Media

🤝 B2B

Senior Researcher exploring models, datasets, and experiments for Troveo’s licensed AI training-data platform. Identifying multimodal data opportunities and guiding product and engineering decisions.

🇺🇸 United States – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

🧬 Research Scientist

Developed by Lior Neu-ner. I'd love to hear your feedback — Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok