Senior Software Engineer, GPU Cluster Infrastructure
Far Ai · Remote (International)
About this role
**Senior Software Engineer, GPU Cluster Infrastructure** **About FAR.AI** FAR.AI is a non-profit AI research institute working to ensure advanced AI is safe and beneficial for everyone. Our mission is to facilitate breakthrough AI safety research, advance global understanding of AI risks and solutions, and foster a coordinated global response. **About the Team — Foundations** Foundations is FAR.AI’s infrastructure and engineering team. We run the compute platform, build the tools and frameworks researchers use, automate research workflows, and help teams scale experiments. We operate a GPU cluster fleet across a mix of bare-metal and managed Kubernetes clusters from multiple providers. The fleet has grown from dozens to hundreds of GPUs and continues to expand—adding providers, onboarding users beyond our researchers, and moving experiments onto frontier open-weight models. We’re standing up an infrastructure sub-team that owns the cluster fleet, including capacity, networking and storage, infrastructure-as-code, and the security posture—plus some platform layer above it. --- ## About the Role You’ll work across the infrastructure stack—from scheduling and storage to monitoring and security—and bring real depth in at least one area. We’re especially interested in experience with: - Large-scale pre-training and post-training infrastructure - Network fabric under GPU clusters - Cluster security and sandboxing - Distributed storage systems - Batch scheduling for large GPU clusters You’ll work directly with researchers and other engineers to keep large-scale experiments performant and fault-tolerant. --- ## What You’ll Do - **Operate the Kubernetes GPU fleet day to day**: node lifecycle, upgrades, driver/image rollouts, staged changes with safe rollback, and capacity planning. - **Own batch scheduling and multi-tenancy**: queues, quotas, priorities, preemption, gang scheduling, and fair share across research teams. - **Design and run storage under the fleet**: high-performance shared filesystems for datasets/checkpoints, object storage tiers, quotas, and backups. - **Keep multi-node training fault-tolerant**: node health, automated draining, debugging NCCL/fabric issues, tracking stragglers/flaky GPUs, and building checkpoint/restart patterns. - **Harden the platform**: identity/access, network policy, secrets, workload isolation, and sandboxing for AI agents running on the cluster. - **Bring new capacity online**: acceptance-test providers (fabric/NCCL/storage throughput), hold to SLAs, and integrate new clusters via infrastructure-as-code. - **Partner with research teams**: turn recurring infrastructure problems into platform fixes; share on-call rotation, runbooks, and postmortems. --- ## Requirements - **3+ years** in systems/infrastructure engineering on production Linux, running GPU/HPC/large-scale batch platforms, owning at least one system end-to-end (design through operation). - **Production Kubernetes for GPU workloads** with a batch layer (e.g., Slurm, Kueue, Volcano, or similar), including quotas, priority/preemption, and node health. - **Infrastructure as code + observability** for a production fleet (e.g., Terraform/Ansible; Helm/ArgoCD; Prometheus or equivalents). - Strong programming skills in **Python, Go, Rust, or C++** (or similar), with automation/services maintained as shared code. - Clear communication for engineers, researchers, and providers (design docs, incident summaries, escalations). If you meet mos
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.