Research Engineer, Intern – Benchmarking
Mercor · San Francisco
About this role
## About Mercor Mercor’s mission is to organize human intelligence to power the AI economy. We’re a leading AI data company building the layer between human expertise and frontier models. Millions of domain experts on our platform are paid over **$4M/day** to train frontier AI models. Our **APEX benchmark** family measures AI’s real-world impact on professional work. Mercor Enterprise brings this infrastructure to Fortune 500 companies—capturing how their best people actually work and translating that expertise directly back into agents. We’re a profitable **Series C** company valued at **$10B**, and we work **in-person five days a week** in **San Francisco, NYC, or London**. --- ## About the Role (Research Engineer, Intern – Benchmarking) As a **Research Engineer**, you’ll work at the intersection of engineering and applied AI research. You’ll own **benchmarking pipelines**, **evaluation systems**, and **failure analysis workflows** that directly inform how we train and improve frontier language models. Your work will help define how we measure **tool use**, **agentic behavior**, and **real-world reasoning**—turning evaluation insights into actionable improvements for **post-training, RLVR, and data pipelines**. --- ## What You’ll Do - **Benchmarking:** Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning; ensure benchmarks scale with training and stay aligned with product and research goals. - **Evaluation systems:** Build and operate end-to-end LLM evaluation systems (runs, scoring, dashboards, reporting) so teams can track performance and compare runs at scale. - **Failure analysis:** Run systematic failure analysis on model outputs (e.g., wrong tool use, reasoning errors, safety/alignment issues); categorize failure modes, quantify prevalence, and feed findings into reward design, data curation, and benchmark design. - **Rubrics and evaluators:** Create and refine rubrics, automated evaluators, and scoring frameworks; balance rigor with scalability (human vs. model-as-judge, calibration, agreement). - **Data quality and usability:** Quantify data usability, quality, and impact on key benchmarks; use evals and failure analysis to guide data generation, augmentation, and curation. - **Cross-team collaboration:** Partner with AI researchers, applied AI teams, and data producers to align evals with training objectives and prioritize the most important benchmarks and failure analyses. - **Ownership in a fast-paced environment:** Operate in a high-iteration research setting with strong ownership of benchmarks, evals, and failure-analysis workflows. --- ## What We’re Looking For - Strong applied research background in **model evaluation, benchmarking, and/or failure analysis**. - Strong coding skills with hands-on experience with **ML models** and **evaluation code**. - Solid grasp of **data structures, algorithms, and backend systems**. - Comfort with **APIs**, **SQL/NoSQL**, and **cloud platforms** for running and storing eval results. - Ability to reason about **model behavior**, **experimental results**, and **data quality** from evals and failure analyses. - Excitement to work **in-person** in **San Francisco** **five days a week** in a high-intensity, high-ownership environment. --- ## Nice to Have - Industry experience on a **post-training** or **evaluation/benchmarking** team (highest priority). - Publications at top-tier venues (NeurIPS, ICML, ACL), especially
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.