Site Reliability Engineer - Fleet Command & Control
Etched · San Jose
About this role
## About Etched Etched is building hardware for frontier intelligence—co-designing chips, racks, software, and manufacturing to deliver best-in-class throughput and latency across prefill and decode workloads. Backed by hundreds of millions from top-tier investors, Etched is redefining the infrastructure layer for the fastest growing industry in history. ## Job Summary You’ll be one of Etched’s first **Site Reliability Engineers**, building the **command-and-control layer** for a fleet of frontier inference systems—covering **alerting, automated recovery, fleet data, and routing** so each problem reaches the right expert. Supercomputing and Infrastructure teams build the fleet management, rollout, and telemetry backbone. You’ll define what signals indicate a healthy system, set how we alert and respond, and build agentic hardware health workflows on top. Fleet SREs own **detection, first response, and handoff**; domain SREs embedded in each team own the deep product fixes. ## Key Responsibilities - **Define what a healthy inference system looks like** - Identify signals that indicate an unhealthy site, rack, node, or chip. - Refine signals as the fleet changes. - Decide which actions run automatically (e.g., drain, cordon, restart, reroute, roll back) vs. which require human paging. - **Set a common alerting standard** - Define SLOs (e.g., time to first token, error rate, availability). - Alert on user-facing symptoms. - Ensure the metrics behind each alert match the alert’s assumptions. - **Cut alert noise** - Suppress expected alerts during deploys and maintenance. - Correlate related alerts into incidents. - Retire pages nobody acts on. - **Automate recovery** - Turn runbooks into agentic hardware health recovery (restarts, drain/traffic shift, spare swaps). - Add guardrails (scoped permissions, rate limits, blast-radius checks, audit trail). - **Route every event to the right engineer** - Classify events by domain. - Page the right domain SRE with full context. - Dispatch on-site technicians for physical work (tray swaps, reseats, leak response). - **Shape the fleet data** - Work with Supercomputing and Infrastructure on telemetry. - Set tracing standards (OpenTelemetry) for end-to-end service visibility. - Report uptime and reliability trends to guide engineering and capacity decisions. - **Run operationally excellent incident management** - Own severity levels, incident channels, communications, postmortems, and corrective-action tracking. - Design on-call rotations, handoffs, and onboarding for sustainability. - **Raise the reliability bar for services** - Review designs for failure modes and graceful degradation. - Partner on canaries, rolling deploys, and graceful drain so updates don’t cause downtime. - **Share on-call in a follow-the-sun rotation** - Work with SREs in other time zones. - **Hands-on debug (first line of defense)** - Acknowledge, size user impact, open incidents, and decide quickly between restart, drain, rollback, and escalation. - **Find where requests are stuck** - Trace latency/throughput regressions through the serving stack (queueing, prefill, decode, handoffs, cache). - **Triage hardware on live racks** - Investigate issues like PCIe drop-offs, firmware panics, chip-to-chip link errors, HBM ECC errors, and power faults. - Run recovery tools and decide when a node should come out. - **Correlate across layers** - Use distributed traces (
Listing freshness
CronJobs last confirmed this listing 56m ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.