Supercompute Operations Engineering
xAI · Southaven, MS; Memphis, TN
About this role
**ABOUT THE ROLE** Own 100% GPU uptime end-to-end, 24/7, across both compute and facility. You’ll lead incident response and the engineering behind it, maintain central control authority from customer impact back to the facility, run day-to-day sustainment of live datahalls, and drive deployment engineering for new datahall bring-up. You’ll also build automation to continuously improve response. You’re expected to have system-level knowledge across the entire datacenter site (end to end) and engineer solutions that enable GPU uptime—working directly with technicians in the datahalls and in the field to execute engineering plans. **RESPONSIBILITIES** - Own monitoring architecture, alerting, signal quality, and automation across GPU/host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU / direct liquid cooling). - Own compute hardware change incorporation (physical, firmware, software). - Participate in 24/7 incident response, mitigation, and end-to-end control authority—from customer impact back to the facility. - Own day-to-day site sustainment operations, resolving issues impacting rack-level compute and facility infrastructure. - Disposition daily issues across Site Operations (rack compute) and Facility Operations (power and cooling). - Lead acceptance and deployment testing campaigns to bring new data halls fully online. - Develop production automation for new datahall bring-up, including automation behind alerting and automated incident response. - Develop and maintain standard operating procedures and planning for facility and site technicians, including change/upgrade incorporation at both rack and facility levels. - Own and document the live, changing state of datahall operations as new halls come online. - Enable Site Operations and Facility Operations technicians to work more efficiently and safely to increase GPU uptime. - Create predictive and responsive automation to make alert/incident response faster and more impactful. - Lead cross-functional reliability projects spanning compute, network (InfiniBand/RoCE), storage, and facility signal boundaries—especially “grey failures” that cross those boundaries (e.g., NVLink flapping, thermal throttling, power sequencing). - Perform true root-cause analysis across systemic/design failure classes across hardware and integrated facility/operations systems; turn fleet data (MTBF/AFR, repeat-RMA trends) into proactive alerting, standard responses, vendor RCA inputs, and supplier corrective actions. **BASIC QUALIFICATIONS** - Bachelor’s degree in Mechanical, Electrical, or Computer Engineering (or related scientific discipline) and 2+ years of experience in data center, critical facilities, power plant, or large-scale compute operations; **or** 6+ years of equivalent experience in those environments. **PREFERRED SKILLS AND EXPERIENCE** - Hands-on experience with at least one: Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry & controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet). - Experience in datacenter site reliability, systems engineering, plant operations, or large-scale production operations. - Experience running acceptance test campaigns for integrated products. - Experience in AI/ML infrastructure or supercomputing environments (e.g., GB200/NVL72 or H100/H200-class GPU fleets), NCCL, InfiniBand/RoCE, Slurm or Kubernetes. - Fleet firmware and BMC management (Redfish, IPMI), including firmware release validation and staged r
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.