Software Engineering Director, Agentic Evaluations

🔥 15 hours ago

🇺🇸 United States – Remote

💵 $260k - $320k / year

⏰ Full Time

🔴 Lead

👮‍♀️ Software Engineering Manager

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 1%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

G2

WebsiteLinkedInAll Job Openings

501 - 1000 employees

Founded 2012

💼 Consulting

📣 Marketing

📦 Logistics

Consulting • Marketing • Logistics

G2 is a business software review platform that provides insights to potential buyers through over 2. 5 million real user reviews. The platform covers a wide range of software categories including project management, video conferencing, e-commerce platforms, CRM, AI chatbots, and many more. G2 empowers businesses to discover and compare software options to make informed purchasing decisions. It serves as a resource for marketers, sales professionals, IT and finance departments to optimize their software investments. Additionally, G2 offers solutions for software vendors to enhance their profiles and reach more buyers.

📋 Description

• Scope feasibility and effort for evaluating software-vendor agents • Own the full stack of the core evaluation system, including admin and API surfaces and evaluation workflow dispatch • Design reusable evaluation system primitives across software verticals • Turn integration onboarding and maintenance processes into repeatable AI skills or agents • Monitor emerging agent-evaluation frameworks and techniques • Establish industry-accepted methods for distributing evaluation results • Collaborate with data science peers to develop proprietary benchmarks • Lead, mentor, and provide technical leadership to a core engineering team • Promote the use of evaluations in agent-oriented products across the organization • Deliver useful and credible evaluations of live software-vendor agents • Apply agent-first engineering techniques to deliver work to production

🎯 Requirements

• 10+ years of professional programming experience in backend or full-stack environments • 2+ years of experience directly managing engineers • Expert-level backend development proficiency using Python, Java/Kotlin, TypeScript/JavaScript, or Go • Strong proficiency with backend frameworks such as FastAPI or Node.js • Direct experience creating evaluations against customer-facing agents using agent trajectory trace data and rubrics • Experience measuring task completion rate, accuracy, correctness, or policy adherence • Direct experience using frontier models from OpenAI, Anthropic, or Google in LLM-as-a-judge applications • Regular use of coding agent harnesses such as Claude Code, Codex, Opencode, or Pi • BS/BA degree in a related field • Experience with agent tool use via direct integration or MCP servers is advantageous • Knowledge of STATE-Bench, tau2-bench, or similar benchmark frameworks is advantageous • Experience with Playwright, browser-use, Chrome DevTools MCP, or similar tools is advantageous • Experience with durable execution/workflow frameworks or agent sandboxing is advantageous • Authorization to work in the United States without restriction • No current or future U.S. work sponsorship requirement

🏖️ Benefits

• Equity • Bonus • Flexible work • Ample parental leave • Unlimited PTO • Inclusive and diverse work environment • Employee resource groups (ERGs) • G2 Gives philanthropic program • AI-assisted application review opt-out option