Sr. Software Engineer - Ingestion Core team
Databricks · San Francisco, California
About this role
**Sr. Software Engineer — Ingestion Core team** Databricks is building the Data + AI platform that helps customers gain insights and drive innovation. A key part of enabling this is making **data ingestion** seamless—turning complex, high-volume data from many sources into reliable, efficient ingestion into **Delta Lake**. ## What you’ll do - Build distributed infrastructure to ingest data from diverse sources, including **streaming ingestion**, **incremental processing**, and **replication**. - Reduce **end-to-end latency**, increase **throughput**, and lower **costs** from source systems to availability in **Delta Lake**. - Design and optimize streaming/distributed workloads for **throughput, cost, latency, reliability, and scale**. - Optimize streaming workloads by exploring and applying **ML techniques**. - Build **monitoring and observability** (customer-facing and internal) to provide visibility into ingestion workflows and systems. - Collaborate with partner teams to enable use cases like **RAG** and **AI agents**. - Build **agentic SDKs** with strong evaluations. ## Data & technologies you’ll work with - Sources: **cloud storage** (SQS, ADLS, GCS), **databases** (Oracle, SQL Server, MySQL, Postgres), and **file sources** (Google Drive, SharePoint) - Formats: structured (**JSON, Parquet, CSV**) and unstructured (**text, images, docs, PPTs, blobs**) - Landing in **Delta Lake** with **schema evolution** and **CDC (change data capture)** capabilities ## Ideal qualifications - **5+ years** writing production code in **Java, Scala, Go, C++, or Python** - Experience architecting, developing, and deploying **large-scale distributed and asynchronous systems** - Experience with **distributed systems, streaming, Spark, databases, data processing, or CDC** - Experience building/operating systems where **scale, throughput, latency, reliability, and cost** matter - Comfort using **AI tools** and defining effective **evaluations (evals)** to speed up development ## Pay range (local) **$166,000 — $225,000 USD** ## About Databricks Databricks is the Data and AI company powering data and AI apps, analytics, and agents for 20,000+ organizations worldwide. *Benefits and diversity/inclusion commitments are provided by Databricks and vary by region.*
Listing freshness
CronJobs last confirmed this listing 56m ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.