Software Engineering Director, Agentic Evaluations

🕒 September 23

🇺🇸 United States – Remote

💵 $260k - $320k / year

⏰ Full Time

🔴 Lead

👮‍♀️ Software Engineering Manager

Where this job comes from

Posted on G2's careers page ↗G2's careers page, not through a job board

First seen by Remote Rocketship September 23

Still open on G2's site. Checked 6 hours ago.

Apply Now
Logo of G2

G2

501 - 1000 employees

Founded 2012

💼 Consulting

📣 Marketing

📦 Logistics

Consulting • Marketing • Logistics

G2 is a business software review platform that provides insights to potential buyers through over 2. 5 million real user reviews. The platform covers a wide range of software categories including project management, video conferencing, e-commerce platforms, CRM, AI chatbots, and many more. G2 empowers businesses to discover and compare software options to make informed purchasing decisions. It serves as a resource for marketers, sales professionals, IT and finance departments to optimize their software investments. Additionally, G2 offers solutions for software vendors to enhance their profiles and reach more buyers.

📋 Description

• Scope feasibility and effort for evaluating software-vendor agents • Own the full stack of the core evaluation system, including admin and API surfaces and evaluation workflow dispatch • Design reusable evaluation system primitives across software verticals • Turn integration onboarding and maintenance processes into repeatable AI skills or agents • Monitor emerging agent-evaluation frameworks and techniques • Establish industry-accepted methods for distributing evaluation results • Collaborate with data science peers to develop proprietary benchmarks • Lead, mentor, and provide technical leadership to a core engineering team • Promote the use of evaluations in agent-oriented products across the organization • Deliver useful and credible evaluations of live software-vendor agents • Apply agent-first engineering techniques to deliver work to production

🎯 Requirements

• 10+ years of professional programming experience in backend or full-stack environments • 2+ years of experience directly managing engineers • Expert-level backend development proficiency using Python, Java/Kotlin, TypeScript/JavaScript, or Go • Strong proficiency with backend frameworks such as FastAPI or Node.js • Direct experience creating evaluations against customer-facing agents using agent trajectory trace data and rubrics • Experience measuring task completion rate, accuracy, correctness, or policy adherence • Direct experience using frontier models from OpenAI, Anthropic, or Google in LLM-as-a-judge applications • Regular use of coding agent harnesses such as Claude Code, Codex, Opencode, or Pi • BS/BA degree in a related field • Experience with agent tool use via direct integration or MCP servers is advantageous • Knowledge of STATE-Bench, tau2-bench, or similar benchmark frameworks is advantageous • Experience with Playwright, browser-use, Chrome DevTools MCP, or similar tools is advantageous • Experience with durable execution/workflow frameworks or agent sandboxing is advantageous • Authorization to work in the United States without restriction • No current or future U.S. work sponsorship requirement

🏖️ Benefits

• Equity • Bonus • Flexible work • Ample parental leave • Unlimited PTO • Inclusive and diverse work environment • Employee resource groups (ERGs) • G2 Gives philanthropic program • AI-assisted application review opt-out option

Apply Now

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

🕒 September 23

CLEAResult

1001 - 5000

💼 Consulting

🏥 Healthcare

📦 Logistics

Engineering Director leading East Region engineering teams for CLEAResult, an energy-efficiency company. Overseeing technical delivery, staffing, budgets, business development, and team growth.

🕒 September 22

Upstart

1001 - 5000

🚘 Automotive

💼 Consulting

🏥 Healthcare

Senior Engineering Manager leading ML data infrastructure for Upstart’s AI lending marketplace. Improving data quality, onboarding, and production readiness for machine learning systems.

🕒 September 22

Render

11 - 50

💼 Consulting

📦 Logistics

☁️ SaaS

Senior Engineering Manager leading Render’s cloud onboarding, activation, and AI agent platform teams. Scaling engineering organizations and developer experiences across human and programmatic interfaces.

🕒 September 22

Forma

51 - 200

🏥 Healthcare

💼 Consulting

📦 Logistics

Senior Engineering Manager leading backend platform strategy, reliability, and scalability. Managing engineers building Forma's employee benefits infrastructure.

🕒 September 22

CSC

5001 - 10000

📋 Compliance

🔒 Cybersecurity

Engineering Manager leading Corptax’s GTI team building tax and regulatory compliance software. Guiding architecture, delivery, production health, and hands-on engineering.

Developed by Lior Neu-ner. I'd love to hear your feedback — Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok