Senior AI Developer
ClickUp · USA
About this role
🚀 **Senior AI Developer (Senior AI Engineer) — ClickUp** At ClickUp, we’re building the future of work: a truly converged AI workspace that unifies tasks, docs, chat, calendar, and enterprise search—supercharged by context-driven AI. We’re an AI-native company, and we evaluate AI fluency as part of our hiring process. We’re looking for a **highly skilled Senior AI Engineer** to join our Engineering team with a focus on **AI evaluations**. You’ll own how we measure and improve the quality of ClickUp’s AI—building evaluation systems, the core agent harness, and the AI that performs well on those evals. --- ## Key Responsibilities - **Design, build, and own** ClickUp’s AI evaluation system, including: - Golden datasets - LLM-as-judge and automated scorers - Regression detection - Quality tracking across AI features - **Build and evolve** the core agent harness and the AI that performs well on those evals (partnering with product and engineering to define what “great” looks like) - Create **self-improvement loops** where agents modify other agents/prompts, rerun evals, and iterate automatically - **Benchmark new models** as they’re released; own model selection, routing, and fallback strategies across providers (e.g., OpenAI, Anthropic, Google, open-weight), balancing **quality, latency, and cost** - Architect and implement **scalable AI platform services** for deployment, orchestration, and lifecycle management of LLMs and other AI models - Build and maintain **robust APIs and backend systems** integrating AI-powered features into ClickUp’s core platform - Develop infrastructure for **model serving, monitoring, logging, and automated evaluation** to ensure reliability and performance in production - Drive best practices in **AI privacy, security, and compliance** (e.g., data anonymization, secure handling, regulatory adherence) - Collaborate cross-functionally with **product, frontend, and data** to deliver seamless, reliable, user-centric AI experiences --- ## Qualifications - **5+ years** of software engineering experience, including **2+ years** building production **LLM or ML systems** - Hands-on experience building **LLM evaluation systems in production** (golden datasets, LLM-as-judge, judge calibration, offline/online evals) tied to real product quality - Strong **experiment design and statistical rigor**, including A/B testing and offline vs. online metrics - Proven track record applying LLMs/AI models to **real-world, customer-facing product features** - Experience integrating/managing **multiple LLMs** (benchmarking, routing, cost/latency/quality tradeoffs) - Deep expertise in **backend engineering, distributed systems, and cloud-native** technologies (e.g., Kubernetes, Docker, AWS/GCP/Azure) - Strong programming skills in **Python** (preferred), Go, TypeScript, or similar - Proficiency with orchestration/workflow frameworks (e.g., **LangGraph, Airflow, Ray**, or similar) - Familiarity with **search and retrieval** technologies and integration into AI-driven applications - Excellent collaboration and communication skills - Passion for staying at the forefront of AI and applying new technologies at scale --- ## Nice to Have - Background in **data science, applied science, or ML engineering** - Experience with eval tooling (e.g., **Braintrust, LangSmith, Weave, Arize**, or in-house frameworks) - Experience with **model routing, AI gateways, or inference optimization** --- ## Notes - **#LI-REMOTE #LI-AK2 #LI-CC1**
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.