Sr. Observability Systems Engineer
Backblaze · Remote - US
About this role
**Role Overview** Senior individual contributor role with significant architectural ownership. You’ll set the technical direction for how Backblaze detects, correlates, and responds to infrastructure incidents across our data centers, and mentor other engineers as the platform matures. If you enjoy making complex distributed systems visible—and you’re excited about applying AI and agentic automation to infrastructure operations—this role was built for you. **Key Responsibilities** **Technical Leadership** - Set technical direction and standards for observability across the organization (instrumentation conventions, alerting philosophy, dashboard design patterns). - Mentor engineers on the Observability team and act as a technical resource for SRE, Storage Engineering, and Network Engineering. - Lead design reviews for observability work and represent the team in cross-functional architecture discussions. - Evaluate build-versus-buy decisions for observability and AIOps tooling, including data sovereignty and compliance considerations. - Produce and maintain design documentation, runbooks, and architectural decision records. **Observability Platform** - Own and evolve the metrics pipeline (VictoriaMetrics, Prometheus, and Consul-based service discovery for scrape target automation). - Own and expand the logging stack (OpenSearch, Logstash, Filebeat) across all Backblaze data centers. - Architect and expand distributed tracing with Jaeger and OpenTelemetry, extending coverage beyond current application instrumentation. - Define and maintain Grafana dashboard standards for infrastructure, storage, and application performance visibility. - Drive expansion of internal application telemetry in partnership with application teams. - Define capacity and retention strategy for the telemetry stack as the fleet grows. **Infrastructure & Automation** - Own infrastructure-as-code for observability stack deployments across multiple Kubernetes clusters. - Write and review automated playbooks for configuration management and automation across Debian-based bare metal servers. - Partner with the SRE team on Consul-based service discovery for VictoriaMetrics scrape target automation. - Lead Ceph storage observability (OSD health, performance metrics, and integration with alerting). - Collaborate with Network Engineering on BGP, routing, and network-level observability. **Required Qualifications** - 8+ years in systems engineering, SRE, DevOps, or observability-focused roles, including demonstrated technical leadership. - Deep hands-on experience with Prometheus, VictoriaMetrics, or equivalent (PromQL, Alertmanager, operating at scale). - Experience designing and operating log aggregation pipelines (OpenSearch/Elasticsearch, Logstash, Filebeat, or similar). - Working knowledge of distributed tracing (Jaeger, OpenTelemetry, or equivalent), including instrumentation strategy. - Proficiency in Python and/or Go for automation and tooling. - Experience with Kubernetes (deploying workloads; understanding pod networking and storage). - Experience with infrastructure-as-code and configuration management. - Strong understanding of Linux internals (Debian/Ubuntu preferred). - Demonstrated experience mentoring engineers and raising the technical bar. - Excellent written and verbal communication; ability to write design documents that drive decisions. **Preferred Qualifications** - Experience leading evaluation/deployment of AIOps tooling (alert correlation, dedu
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.