Software Engineer, Applied AI- Evals
Mercor · San Francisco
About this role
## About Mercor Mercor’s mission is to organize human intelligence to power the AI economy. We’re a leading AI data company building the layer between human expertise and frontier models—where millions of domain experts are paid to train models. Mercor’s **APEX benchmark family** measures real-world impact on professional work. **Mercor Enterprise** brings this infrastructure to Fortune 500 companies, capturing how top people actually work and translating that expertise back into agents. We’re a profitable **Series C** company valued at **$10B**, and we work **in-person five days a week** in **San Francisco, NYC, or London**. --- ## About the Role (Software Engineer, Applied AI — Evals) Mercor’s **evals platform** helps frontier labs and enterprises understand how their models perform on real work. It runs evals at scale, grades results, and explains why models fail. As a Software Engineer on the **Evals product team**, you’ll build the scorecards, failure reports, and dataset views that help customers turn eval runs into decisions. You’ll solve hard problems in scale, grading accuracy, and reproducibility—while keeping the product simple to use. This role can grow from evals into post-training and new product areas, helping turn failures into data that improves models. --- ## What You’ll Do - **Build the product customers use to evaluate models**: Ship APIs, data models, and UI to run evals, compare models, and share scores. - **Own APEX launches**: Run every new frontier model on APEX and publish correct, defensible results to the public leaderboard. - **Explain why models fail**: Build failure analysis that links failure patterns to the exact steps that went wrong. - **Make benchmarks easy to find and manage**: Create dataset views, registries, and versioning for benchmark discovery and management. - **Take customers from first login to self-serve**: Build onboarding, roles, key management, and usage views for teams to run evals independently. - **Work directly with customers**: Join pilots with labs and enterprises and translate needs into the roadmap. --- ## What We’re Looking For - **B.S. in Computer Science** (or related field) - Strong full-stack skills using a modern backend language (**Python, Go, or TypeScript**) and a frontend framework such as **React** - Experience **owning product features end-to-end**, from idea to production - Strong **product sense** focused on user needs - Experience building **data-heavy products** (data models, APIs, and views over large volumes of results) - Excellent **written and verbal communication** with engineers, researchers, and customers - Comfort with **ambiguity** and a habit of **shipping iteratively** - **Plus**: Experience with LLM evals, benchmarks, or AI developer tools - **Plus**: Startup experience and/or building **B2B developer products** --- ## Benefits - Bi-annual performance bonus structure - Generous equity grant vested over 4 years - Up to **$15k relocation bonus** - **$10K housing bonus** (if you live within 0.5 miles of the office) - **$1.5K monthly stipend** for meals - Free **Equinox membership** - **$200 monthly laundry reimbursement** - **$200 monthly personal wellness reimbursement** - Health, Dental, Vision insurance
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.