Senior Software Engineer, Backend (Workload Safety)
Affirm · Remote US
About this role
**Affirm — Senior Software Engineer, Backend (Workload Safety)** Affirm exists for the moments that matter—giving people a clear, predictable way to pay over time, with no hidden fees, no surprises, and no tradeoffs on what matters most. Affirm’s engineering team builds large-scale, highly available, global infrastructure shared across multiple financial products. Ensuring this infrastructure is accessible to all engineers is critical to the business. The Workload Safety team creates standards for how workloads are configured and run across Affirm’s **85+ Amazon EKS clusters** (and growing). The team identifies industry best practices, drives adoption across engineering teams, and enforces compliance through **policy and automation**. As a senior engineer, you’ll help set the team’s roadmap, decide which standards matter most, and build a repeatable path from research to enforcement across the fleet. --- ### What you’ll do - Define and evolve workload safety standards for Affirm’s EKS fleet (e.g., CPU/memory settings, workload isolation/placement, disruption tolerance, graceful termination). - Own each standard end to end: research/prototyping, onboarding for early adopters, self-service guidance for everyone else, and automation/policy guardrails to ensure compliance from day one. - Lead cross-team initiatives from design through fleet-wide rollout; mentor engineers; set a high bar for design, code review, and operational readiness. - Isolate workloads by criticality and resource profile so one misbehaving workload can’t degrade critical paths like checkout. - Fix stuck workloads and node lifecycle problems (e.g., pods that ignore termination signals, PDBs blocking node cycling, Karpenter timeouts preventing bin packing/removal of excess capacity). - Determine safe operating limits per cluster (pod count, node count, API server request volume) and develop control plane scaling patterns for bursty workloads (scheduled batch jobs, large deployments). - Analyze workload resource profiles across the fleet (CPU, memory, disk, network) to guide EC2 instance type strategy, and automate analysis as workloads change. - Measure each standard’s impact on availability, latency, risk, and cost to prioritize the roadmap. - Participate in on-call for Kubernetes runtime issues across the fleet, and turn recurring failure patterns into roadmap items. - Work closely with teams running services, batch jobs, and ML workloads on EKS, plus deployment/capacity/SRE teams. - Stay current with Kubernetes community best practices and adapt them to Affirm’s needs. --- ### What we look for - **5+ years** running Kubernetes in production at scale (ideally Amazon EKS across many clusters). - Deep understanding of Kubernetes resource management and scheduling (requests/limits, OOM and CPU throttling behavior, affinity/anti-affinity, taints/tolerations, topology spread constraints). - Solid grasp of pod lifecycle and disruption management (graceful termination, pod disruption budgets, node draining). - Experience enforcing configuration standards via admission control and policy-as-code (e.g., **OPA Gatekeeper** or **Kyverno**). - Proficiency in **Go or Python**, including building production infrastructure software (e.g., controllers, operators, admission webhooks). - Strong production debugging skills and a track record of turning recurring incidents into systemic fixes. - Experience leading technical initiatives across multiple teams, driving adoption of sha
Listing freshness
CronJobs last confirmed this listing 2h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.