Senior AI Developer
ClickUp · United States
About this role
**About ClickUp** At ClickUp, we’re building the future of work: a truly converged AI workspace that unifies tasks, docs, chat, calendar, and enterprise search—supercharged by context-driven AI. We’re an AI-native company, and we evaluate AI fluency as part of our hiring process. 🚀 --- **Role: Senior AI Developer (Senior AI Engineer) — AI Evaluations** Join ClickUp’s Engineering team to help redefine what’s possible by owning how we measure and improve the quality of ClickUp’s AI. You’ll build evaluation systems, the core agent harness, and the AI that performs well on those evaluations—while working hands-on with LLMs to solve real user problems. --- **Key Responsibilities** - Design, build, and own ClickUp’s AI evaluation system (golden datasets, LLM-as-judge, automated scorers, regression detection, and quality tracking). - Build and evolve the core agent harness and the AI that performs well on those evals—partnering with product and engineering to define what “great” looks like. - Create self-improvement loops where agents modify other agents/prompts, rerun evals, and iterate automatically. - Benchmark new models and own model selection, routing, and fallback strategies across providers (e.g., OpenAI, Anthropic, Google, open-weight), balancing quality, latency, and cost. - Architect and implement scalable AI platform services for deployment, orchestration, and lifecycle management of LLMs and other AI models. - Build and maintain robust APIs and backend systems integrating AI-powered features into ClickUp’s core platform. - Develop infrastructure for model serving, monitoring, logging, and automated evaluation to ensure reliability and performance in production. - Drive best practices in AI privacy, security, and compliance (data anonymization, secure handling, regulatory adherence). - Collaborate cross-functionally with product, frontend, and data teams to deliver seamless, reliable, user-centric AI experiences. --- **Qualifications** - 5+ years of software engineering experience, including 2+ years building production LLM or ML systems. - Hands-on experience building LLM evaluation systems in production (golden datasets, LLM-as-judge, judge calibration, offline/online evals) tied to real product quality. - Strong experiment design and statistical rigor (A/B testing, offline vs. online metrics, separating true regressions from noise). - Proven track record applying LLMs/AI models to real, customer-facing product features. - Experience integrating and managing multiple LLMs (benchmarking, routing, and cost/latency/quality tradeoffs). - Deep backend engineering expertise (distributed systems, cloud-native technologies such as Kubernetes, Docker, AWS/GCP/Azure). - Strong programming skills in Python (preferred), Go, TypeScript, or similar. - Proficiency with orchestration frameworks/workflow engines (e.g., LangGraph, Airflow, Ray, or similar). - Familiarity with search and retrieval technologies and their integration into AI-driven applications. - Excellent collaboration and communication skills. - Passion for staying at the forefront of AI and applying new technologies at scale. --- **Nice to Have** - Background in data science, applied science, or machine learning engineering. - Experience with eval tooling (e.g., Braintrust, LangSmith, Weave, Arize, or in-house frameworks). - Experience with model routing, AI gateways, or inference optimization. --- **Location / Tags** - #LI-REMOTE - #LI-AK2 - #LI-CC1 --- **Equal
Listing freshness
CronJobs last confirmed this listing 4h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.