Researcher, AI Evaluations
Figma · San Francisco, CA • New York, NY • United States
About this role
**Figma — Researcher, AI Evaluations** Figma is growing our team of passionate creatives and builders on a mission to make design accessible to all. Figma’s platform helps teams bring ideas to life—whether you’re brainstorming, creating a prototype, translating designs into code, or iterating with AI. We’re hiring a **Researcher, AI Evaluations** to join **AI Insights** within Figma’s **Research** team. This is a **senior individual contributor** role with **no direct reports**, reporting to the **AI Insights lead** and working directly with **AI Research** within **AI Platform**. You’ll strengthen the team’s model, agent system, and evaluation work through hands-on studies, clear rubrics, and reliable measures of quality that reflect user needs—guiding model and product improvements. **Location**: Full-time role, held from one of our **US hubs** or **remotely in the United States**. --- ## What you’ll do at Figma - **Own evaluation research** for priority AI Research workstreams: define quality criteria, design studies, analyze results, and recommend actions. - **Build rubrics, evaluation datasets, benchmarks, and quality thresholds** that reflect user needs and expert design judgment. - **Partner with AI Research and Modeling Platform engineers** to integrate validated methods into repeatable evaluations and regression tests. - **Explain quality, failure patterns, and uncertainty**, and recommend actions that guide model selection, model improvements, product priorities, and release decisions. - **Validate automated graders** against human ratings, including **LLM-as-judge** methods to score outputs; identify bias, gaps, and uncertainty. - **Connect user research with AI Research’s model development**, sharing reusable methods, reviewing work, and mentoring peers. --- ## We’d love to hear from you if you have - **7+ years** of experience in research, applied research, or a related field, with a proven track record of independently leading complex evaluation projects and delivering measurable impact. - **Hands-on experience evaluating AI products**, with demonstrated ability to translate findings into meaningful model and product improvements. - Expertise designing **human evaluations**, **scoring rubrics**, **datasets**, and **benchmarks**—including measuring **inter-rater agreement** and validating automated graders. - Strong **qualitative and quantitative** research skills, including analyzing data, interpreting uncertainty, and drawing actionable insights about model behavior and performance. - Excellent **problem-solving** and **stakeholder management** skills—navigating ambiguity, prioritizing risks, balancing speed with rigor, and delivering recommendations that influence cross-functional decisions. --- ## Added plus (not required) - Experience integrating automated evaluations and regression testing into engineering workflows (e.g., **Braintrust, LangSmith, DeepEval**, or similar). - Ability to develop scalable, reusable evaluation frameworks and drive adoption of consistent quality standards. - Background in **product design, product management, data science, or engineering**, with familiarity in **Figma** or similar design/developer tools. - Strong statistical expertise in **sampling, significance testing, and agreement metrics**, including experiments or analysis of AI feature performance. - Advanced training in a quantitative or behavioral discipline (formal education or equivalent hands-on experience). --- ## Pa
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.