Tech Lead Manager, GPU Cluster Infrastructure
Far Ai · Remote (International)
About this role
**Tech Lead Manager, GPU Cluster Infrastructure** ## About FAR.AI FAR.AI is a non-profit AI research institute working to ensure advanced AI is safe and beneficial for everyone. Our mission is to facilitate breakthrough AI safety research, advance global understanding of AI risks and solutions, and foster a coordinated global response. We conduct pre-deployment testing, run independent evaluations for governments, and publish the AI Security Leaderboard based on our red-teaming expertise. ## About the Team: Foundations Foundations is FAR.AI’s infrastructure and engineering team. We run the compute platform, build the tools and frameworks researchers use, automate research workflows, and help teams scale experiments. We operate a growing fleet of bare-metal and managed Kubernetes GPU clusters across multiple providers. The platform supports frontier open-weight model research and increasingly, agent-driven experiments that require strong infrastructure performance and security. We’re standing up an **infrastructure sub-team** that owns the cluster fleet—capacity planning, networking and storage design/management, infrastructure as code, and security posture—working closely with research teams as their needs evolve. ## About the Role You’ll be the **lead of the infrastructure sub-team** and one of its engineers, with **at least half your time on technical work**. You’ll own the platform’s technical direction and roadmap, decide what we build and how we run it in collaboration with research teams and the rest of Foundations, and **hire and grow** the team. In frontier AI research, infrastructure is often part of the science—so you’ll work directly with researchers and engineers to keep large-scale experiments **performant and fault-tolerant**. > Note: We’re also hiring a **Software Engineer, GPU Cluster Infrastructure** for hands-on work without the management responsibilities. ## What You’ll Do - **Set the platform’s technical direction** and own its roadmap (systems we run, scheduling/storage across providers, and key metrics like utilization, queue wait, and failure rates). - **Stay hands-on**: own architecture plus scheduling/storage design, and debug cross-layer failures (node health, GPU/fabric faults, multi-node job hangs). - **Hire and grow a small team** of senior engineers: set priorities, scope projects, and provide regular feedback/coaching. - **Set security direction** for a shared cluster used by AI agents (identity/access, workload isolation, sandboxing). - **Define and participate in operations**: on-call/incident response approach, postmortems, fault tolerance, and observability. - **Escalation point** for research teams and providers; turn recurring issues into platform improvements. ## Requirements - Experience leading engineers as a **manager/tech lead/project lead**, setting technical direction, scoping work, and giving feedback. You want to manage people directly (prior direct reports not required). - **5+ years** in production systems/infrastructure engineering on Linux, running GPU/HPC/large-scale batch platforms; owned systems from design through operation. - Depth in at least one area: **scheduling, storage, networking, security, or GPU systems**, plus enough breadth to review other designs. - Experience running **production Kubernetes for GPU workloads** with a batch layer (e.g., Slurm, Kueue, Volcano), including quotas, priority/preemption, and node health. - Owned **infrastructure as code** and **obs
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.