Principal Research Scientist

πŸ•’ October 27

πŸ‡ΊπŸ‡Έ United States – Remote

πŸ’΅ $190k - $235k / year

⏰ Full Time

🟑 Mid-level

🟠 Senior

🧬 Research Scientist

Where this job comes from

Posted on G2's careers page β†—G2's careers page, not through a job board

First seen by Remote Rocketship October 27

Still open on G2's site. Checked 2 hours ago.

Apply Now
Logo of G2

G2

501 - 1000 employees

Founded 2012

πŸ’Ό Consulting

πŸ“£ Marketing

πŸ“¦ Logistics

Consulting β€’ Marketing β€’ Logistics

G2 is a business software review platform that provides insights to potential buyers through over 2. 5 million real user reviews. The platform covers a wide range of software categories including project management, video conferencing, e-commerce platforms, CRM, AI chatbots, and many more. G2 empowers businesses to discover and compare software options to make informed purchasing decisions. It serves as a resource for marketers, sales professionals, IT and finance departments to optimize their software investments. Additionally, G2 offers solutions for software vendors to enhance their profiles and reach more buyers.

πŸ“‹ Description

β€’ Own the evaluation methodology and set standards for evaluating AI agents β€’ Work with engineers and product managers to build and improve evaluations in G2’s current offerings β€’ Scope features, turn methods into production-ready components, and iterate based on customer and business needs β€’ Design realistic environments and task sets reflecting business work β€’ Develop accurate, reproducible and difficult-to-game scoring approaches using automated checks, LLM-based judges and human review β€’ Measure consistency and reliability of evaluations β€’ Audit tasks and graders, quantify noise, and establish when agent differences are real β€’ Present and defend evaluation findings to buyers, customers, partners and the research community β€’ Mentor and guide researchers and engineers on evaluation design and measurement β€’ Review work and document standards for repeatable evaluations β€’ Set and pursue a research agenda across agent evaluation, user simulation, LLM-as-judge reliability, voice and multimodal agents, long-horizon workflows and test-set contamination β€’ Follow new benchmarks, papers and evaluation practices and translate them into product and roadmap decisions β€’ Write methodology documentation, papers and public posts where appropriate β€’ Partner with data and go-to-market teams on presentation and use of results

🎯 Requirements

β€’ Experience working with evaluation design or dataset curation in multi-agentic systems β€’ 5+ years building or evaluating ML systems, including hands-on work with LLMs or LLM-based agents β€’ 2+ years experience managing small to medium teams preferred β€’ A graduate degree (MS or PhD) in machine learning, NLP, computer science, statistics or a related field, or equivalent applied research experience β€’ Track record of original ideas in machine learning, shown through first-author publications or projects led end-to-end β€’ Ability to own a research agenda and carry long-running projects independently β€’ Track record of shipping ML work in a product setting with engineers and product managers β€’ Experience setting technical direction, mentoring researchers or engineers, and reviewing methodology β€’ Familiarity with AI benchmarking and benchmark limitations β€’ Sound statistical judgment, including experimental design, variance, significance and agreement between human and automated raters β€’ Good judgment about practical trade-offs β€’ Clear writing and ability to explain methodology to non-specialists and defend it to skeptical audiences β€’ Experience in agent evaluation, user simulation, or human and AI-based judging is helpful β€’ Experience with conversational, voice, retrieval-based or tool-using AI systems is helpful β€’ Contributions to open-source evaluation tools or public leaderboards are helpful β€’ Familiarity with business software workflows is helpful β€’ Publications at ACL, EMNLP, NeurIPS, ICML, ICLR, COLM or similar venues are helpful

πŸ–οΈ Benefits

β€’ Equity β€’ Flexible work β€’ Ample parental leave β€’ Unlimited PTO

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ October 26

SandboxAQ

51 - 200

πŸ€– Artificial Intelligence

πŸ”’ Cybersecurity

πŸ’Š Pharmaceuticals

Principal Scientist leading AI-driven discovery of rare-earth-lean permanent magnets. Guiding simulation, experimental validation, IP, and U.S. manufacturing partnerships.

πŸ•’ October 23

GondolaBio

11 - 50

πŸ₯ Healthcare

πŸ’Ό Consulting

🏭 Manufacturing

Translational medicine scientist advancing biomarker strategies for GondolaBio’s genetic-disease therapeutics. Leading assay validation and clinical implementation across early neuroscience programs.

πŸ‡ΊπŸ‡Έ United States – Remote

πŸ’΅ $190k - $210k / year

πŸ’° $300M Venture Round - GondolaBio on 2024-08

⏰ Full Time

🟠 Senior

🧬 Research Scientist

πŸ•’ October 20

Zillow

5001 - 10000

🏠 Real Estate

πŸ›οΈ eCommerce

πŸ‘₯ B2C

Applied Research Scientist building Zillow’s AI World Model for housing-market demand and planning. Developing uncertainty-aware models that power next-generation home-shopping experiences.

πŸ•’ October 20

Pindrop

201 - 500

πŸ›‘οΈ Insurance

πŸ’Ό Consulting

πŸ₯ Healthcare

Research Scientist advancing AI-driven voice, fraud, identity, and deepfake security at Pindrop. Developing deep learning models and translating research into production-ready enterprise solutions.

πŸ•’ October 20

Troveo AI

11 - 50

πŸ€– Artificial Intelligence

πŸ“± Media

🀝 B2B

Senior Researcher exploring models, datasets, and experiments for Troveo’s licensed AI training-data platform. Identifying multimodal data opportunities and guiding product and engineering decisions.

Developed byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok