Principal Software Engineer - Observability & Telemetry Data
Smartsheet · Bellevue, WA, USA
About this role
**Principal Software Engineer - Observability & Telemetry Data** **About the Role** Set technical direction for how Smartsheet collects, models, stores, and derives value from telemetry data. This architect role defines the telemetry data platform and standards, carries architecture across engineering org boundaries, and serves as the company's definitive technical voice on observability. **Key Responsibilities** • Architect the end-to-end telemetry data platform (landing, modeling, querying, retention, storage/compute separation) • Define semantic conventions and shared business identifiers for cross-signal correlation • Lead migration to OpenTelemetry-based instrumentation with collector architecture and sampling strategy • Own architecture connecting observability platform to Databricks and MLflow for AI/agentic telemetry • Bring first-class observability to data estate (Databricks jobs, pipelines, warehouses) • Build analytics layer on telemetry for reliability metrics, cost models, and adoption reporting • Architect high-volume collection and routing tier across US, EU, AU, and GovCloud regions • Set cost architecture for observability data with ingest governance and cardinality control • Partner across Data Platform, AI Platform, and infrastructure organizations • Lead design reviews and mentor senior engineers on instrumentation and data modeling • Participate in production support and on-call rotation **Required Experience** • 10+ years building/operating large-scale distributed systems, data platforms, or observability infrastructure (Principal/Staff level) • Deep hands-on production ownership of metrics, logs, and distributed tracing at scale • Large-scale data architecture expertise (Delta Lake, Apache Iceberg, Spark, streaming ingestion, schema evolution) • Practical Databricks experience (jobs, clusters, Unity Catalog, Delta, MLflow) • OpenTelemetry depth (collectors, semantic conventions, context propagation, tail-based sampling) • High-volume log/telemetry pipeline experience (FluentBit, Kinesis/Kafka, search backends) • Advanced AWS & Kubernetes expertise (EKS, ECS Fargate, EC2, Lambda, CloudWatch) • 10+ years programming with Go, Java, Python, or Scala; strong SQL • Infrastructure as Code (Terraform, GitOps with Flux/ArgoCD) • Track record of setting technical direction adopted across multiple teams/organizations • Strong incident response instincts • CS/Engineering degree or equivalent industry experience • Legally eligible to work in the U.S. **Nice to Have** • LLM/agentic system instrumentation experience • Data reliability engineering (freshness, quality, lineage SLOs) • Telemetry cost engineering or FinOps at scale • Regulated environment experience (FedRAMP, GovCloud) • Prometheus, Grafana, Alertmanager • Snowflake or multi-platform lakehouse/warehouse experience • Service catalog/internal developer platform work (Backstage) **Location & Compensation** Bellevue, WA office or remote (US-based, where Smartsheet is registered) **Salary Range:** $222,500 – $257,500 USD + market competitive incentive **Benefits** Medical/vision/dental coverage, 401k match, monthly stipend, flexible time away, life/disability insurance, 12 paid holidays, up to 24 weeks parental leave, volunteer day, professional development, company perks
Listing freshness
CronJobs last confirmed this listing 11h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.