Staff Software Engineer, Backend (Observability)
Affirm · Remote US
About this role
## About Affirm At Affirm, we exist for the moments that matter—giving people a clear, predictable way to pay over time, with no hidden fees, no surprises, and no tradeoffs on what matters most. ## Observability Team The Observability team builds the platform every Affirm engineer relies on to understand the behavior and performance of our systems. Our platform ingests **200+ TB/day** of metrics, logs, and traces from **hundreds of services** across **80+ Kubernetes clusters**. When a user’s checkout fails, our data is how engineers find out why. Over the next year, we’re moving to **wide events** (one context-rich event per request) to improve signal-to-noise and make debugging faster. We’re also introducing **log sampling** and building observability best practices into **agentic development and code review**, so AI-assisted code is well instrumented by default. ## Role: Staff Software Engineer, Backend (Observability) We’re looking for a **Staff Software Engineer** to be the **technical lead for Observability**. You’ll spend most of your time hands-on while defining the roadmap, driving cross-functional projects, mentoring a team of six engineers, and partnering closely with infrastructure and product teams. ### What You’ll Do - Own the architecture of Affirm’s telemetry pipeline, balancing **reliability, query performance, and cost** at **200+ TB/day**. - Design **log sampling** strategies to reduce volume and cost without losing the signal engineers need during incidents. - Shape how observability fits into **agentic development**, so AI coding agents and code review catch instrumentation gaps before code ships. - Set technical strategy for a **year-long time scale** and help your team deliver critical, business-impacting projects. - Collaborate across the product development lifecycle with **product management, design, and analytics** to manage technical sustainability, risks, and trade-offs. - Act as a force-multiplier through definition and advocacy of technical solutions and operational processes. - Own team operations and availability: ensure the right **monitoring, triage rotations, playbooks, policies, testing, and alerting** are in place for “keep the lights on” on-call. - Foster a culture of quality and ownership by setting **code review and design standards**, and advocating beyond your team through writing and tech talks. - Develop talent through feedback, guidance, and leading by example. ### What We Look For - **8+ years** experience designing, developing, and launching backend systems at scale (e.g., **Python** or **Kotlin**). - Extensive track record building highly available distributed systems using **AWS, MySQL, Flink, and Kubernetes**. - Experience building or operating high-volume telemetry systems (metrics/logs/traces) using **OpenTelemetry, Kafka, and Terraform**. - Strong point of view on modern observability practices: **structured and wide events, sampling, cardinality management, and SLOs**. - Ability to drive adoption of platform standards across many teams via tooling and engagement. - Experience delivering major features/components or deprecating functionality through clear technical and execution planning; you write high-quality code others can easily understand and use. - Thrive in ambiguity and can move from low-level implementation details to the architecture of large systems. - Demonstrated growth and impact through gathering and iterating on feedback from engineering and cross-functional peer
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.