Software Engineering Director, Agentic Evaluations
G2 · Remote (US)
About this role
**Software Engineering Director — Agentic Evaluations (G2)** ## About G2 G2 is the world’s largest and most trusted software marketplace, helping businesses make better decisions with trusted insights from real software users. G2 has joined forces with Capterra, SoftwareAdvice, and GetApp to create a centralized source of online data and software insights for the age of AI. ## About the Role As a **Software Engineering Director, Agentic Evaluations**, you’ll lead the delivery of **useful and credible evaluations of agents** offered by software vendors. You’ll enhance the system that validates agent performance and lead the core team behind it—using **agent-first** ways of working and agentic engineering techniques to ship to production. ### Summary of Responsibilities - **Build and enhance agent evaluations** that run against live agents (**80%**) - **Lead and mentor engineers**, and influence how evals are used across the organization (**20%**) ## Detailed Responsibilities ### Build & Enhance Agent Evaluations (80%) - Scope feasibility and effort for evaluating vendor-provided agents, and identify the most practical path to credible evaluations - Own the full stack of the core evaluation system (admin & API surface, eval workflow dispatch) - Design evaluation primitives that generalize across software verticals (so onboarding new categories reuses existing work) - Define repeatable onboarding and integration processes to increase evaluation velocity - Track emerging frameworks/techniques for agent evaluation and help distribute eval results using industry-accepted approaches - Collaborate with data science to develop rich and proprietary benchmarks ### Lead a Group of Engineers (20%) - Facilitate growth and mentorship of a core engineering team - Provide technical leadership and subject matter expertise in evaluations - Help socialize the use of evals in agent-oriented products across the wider organization ## Minimum Qualifications - **10+ years** professional programming experience in backend or full-stack environments - **2+ years** experience directly managing engineers - Expert backend development skills (e.g., **Python, Java/Kotlin, TypeScript/JavaScript, Go**) and strong backend frameworks (e.g., **FastAPI, Node.js**) - Direct experience creating evaluations against a **customer-facing agent**, using agent trajectory trace data and rubrics to measure task completion, accuracy/correctness, or policy adherence - Direct experience using frontier LLMs (e.g., **OpenAI, Anthropic, Google**) in **LLM-as-a-judge** applications - Regular use of coding agent harnesses (e.g., **Claude Code, Codex, Opencode, Pi**) in daily development - BS/BA in a related field ## What Can Help Your Application Stand Out - Experience with agent tool use via direct integration or mediated by **MCP servers** - Knowledge of **STATE-Bench**, **tau2-bench**, or similar benchmark frameworks - Experience using **Playwright**, browser-use, **Chrome DevTools**, **MCP**, or similar to drive browser user flows - Experience with durable execution/workflow solutions (e.g., **Temporal**, **DBOS**, **Cloudflare/Vercel Workflows**) or agent sandboxing (e.g., **AWS E2B**, **Cloudflare/Vercel Containers**) ## Commitment to Inclusivity & Diversity G2 is committed to creating an inclusive and diverse environment where people of every background can thrive. ## Privacy / AI in Hiring G2 may use AI-powered technology to support parts of the hiring process (e.g., initial
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.