Staff Software Engineer - Compiler & Spark Runtime
Snowflake · US-CA-Menlo Park
About this role
## Staff Software Engineer — Compiler & Spark Runtime **Location:** Bellevue, WA or Menlo Park, CA ### About the Team The Spark Runtime Engineering Team brings Snowflake’s performance, efficiency, and reliability to Spark workloads. Our mission is to help customers scale data processing while preserving the familiar Spark APIs and semantics they rely on. We build the compiler and runtime integration that translate Spark workloads into efficient execution on Snowflake’s cloud-native query engine—covering compiler infrastructure, query-plan lowering, Spark compatibility, execution integration, and end-to-end observability. ### Role Overview Own complex projects from design through implementation and production rollout. You’ll contribute directly to the codebase, review critical designs and code, and investigate challenging correctness, performance, and reliability issues. You’ll provide technical leadership for engineers building production compiler and runtime capabilities for Spark workloads on Snowflake—guiding through technical judgment, hands-on execution, mentorship, and influence. ### Responsibilities - Lead technical design and implementation of Spark query compilation and runtime integration (IRs, type resolution, semantic compatibility, plan transformations, SQL generation) - Own complex engineering initiatives end-to-end (problem definition → architecture → implementation → validation → rollout) - Provide technical leadership: clarify trade-offs, align contributors, review designs/code, resolve blockers - Improve end-to-end performance and cost-efficiency via profiling, compilation-latency reduction, metadata-access improvements, and collaboration with query-engine teams - Establish correctness and compatibility practices (differential testing vs. Apache Spark, regression coverage, clear handling of unsupported behavior) - Drive production quality: observability, incident investigation, fault recovery, safe rollout practices - Mentor engineers through design discussions, code reviews, debugging, and knowledge sharing - Partner with senior technical leaders and Product Management to prioritize investments and scope milestones - Validate against representative customer workloads and track measurable improvements (compatibility, latency, reliability, cost-efficiency) ### Minimum Qualifications - 10+ years of hands-on software engineering experience, including substantial work on compilers, query-processing systems, or runtime infrastructure - Proven technical leadership for complex initiatives (guiding delivery without formal management authority) - Track record delivering production systems and owning significant architectural/implementation decisions - Deep expertise in at least one relevant area (compiler architecture/IR transformations, relational query processing, language semantics/type systems, or runtime integration) - Proficiency in Scala, Java, C++, or a comparable implementation language - Strong systems fundamentals (concurrency, memory behavior, performance analysis) and experience debugging complex production issues - Ability to translate technical/customer requirements into practical engineering priorities across teams - Strong technical communication and mentorship skills ### Preferred Qualifications - Experience with Apache Spark internals (Catalyst, Spark Connect, DataFrame APIs) - Experience with relational algebra, SQL generation, query optimization, or database execution systems - Experience with Scala
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.