Staff AI Engineer
Cobalt.io · Remote
About this role
<h2><strong>Description</strong></h2> <p>Cobalt runs AI agents in production that test real customer applications. We’re looking for a Staff AI Engineer who has built and operated production agent systems and can help us make them more reliable, secure and cost-efficient.</p> <p>You’ll set the technical direction for how we design, evaluate, ship and operate agents, and stay hands-on building the foundations our teams use. You’ll work across engineering, product, research and security, mentoring engineers and helping teams turn promising prototypes into dependable production systems.</p> <p>Your primary focus will be our agent platform and production practices. You’ll also help engineering teams improve how they build software with AI coding agents.</p> <h2><strong>What You’ll Do</strong></h2> <ul> <li>Set technical direction for agent systems across teams, including orchestration, tool use, context and memory management, human oversight and failure recovery.</li> <li>Design, build and operate shared foundations: evaluation frameworks, agent observability and tracing, execution controls, and model access with routing and failover where needed.</li> <li>Establish evaluation practices that measure task completion, accuracy, false positives, security and cost per successful task. Work with domain experts to build representative test cases and turn production failures into regression tests.</li> <li>Make agent workflows resilient to tool failures, timeouts, rate limits and partial execution, with clear limits on retries, runtime and spend.</li> <li>Work with security and platform teams to build in scoped permissions, customer data isolation, audit trails and safe execution of tools and code.</li> <li>Establish practical engineering standards: evals in CI, prompt and model versioning, regression testing for non-deterministic systems, cost and latency budgets, and staged rollouts with rollback paths.</li> <li>Lead model selection and cost optimization. Benchmark models against real workloads and make informed trade-offs between quality, latency and cost.</li> <li>Partner with teams on production readiness, incident investigation and improvements to agent reliability.</li> <li>Help teams use AI coding agents effectively across planning, implementation, review and testing. Measure the impact on delivery speed and software quality.</li> <li>Mentor engineers and technical leads through design reviews, pairing, internal talks and written guidance. Help teams adopt shared capabilities and improve them based on feedback.</li> <li>Work with product, research and security leaders to turn business priorities into a technical roadmap with clear outcomes.</li> </ul> <h2><strong>You Must Have</strong></h2> <ul> <li>Significant software engineering experience, typically 8-10+ years, with a track record of technical leadership across teams.</li> <li>Hands-on experience building and operating LLM-based agents in production, with a clear understanding of their scale, constraints and business outcomes.</li> <li>A track record of improving agent reliability: building evaluations, investigating non-deterministic failures and improving success rates against measurable...
Listing freshness
CronJobs last confirmed this listing 57m ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.