Data Platform Engineer
Brightai · Palo Alto, CA
About this role
We are a high-growth company transforming how businesses operate by integrating AI, IoT, and cloud-native services into scalable, real-time platforms. As a **Platform Data Engineer**, you’ll play a critical role in building and maintaining the data infrastructure that powers our products, services, and insights. You’ll join a multidisciplinary team focused on ingesting, processing, and managing massive streams of sensor and operational data across a wide array of devices—from drones and robots to industrial systems and smart environments. **Responsibilities** - Design, build, and maintain scalable, reliable, and high-throughput data ingestion pipelines for structured and semi-structured data. - Implement robust and secure data lake and SQL-based storage architectures optimized for performance and cost. - Develop and maintain internal tools and frameworks for data ingestion using **Python, Golang, and SQL**. - Build and maintain inference pipelines to support model serving and near real-time prediction workflows. - Develop and maintain internal tools and deployment processes to support the ML development workflow, including experimentation, testing, deployment, and monitoring. - Collaborate cross-functionally with Cloud, Edge, Product, and AI teams to define data contracts, schemas, and retention policies. - Use **AWS** cloud infrastructure (including **Argo Workflows, S3, Lambda, Glue, Kinesis, Athena, and RDS**) to support end-to-end data workflows. - Employ **Infrastructure-as-Code (IaC)** practices using **Terraform** to manage data platform infrastructure. - Monitor data pipelines for quality, latency, and failures using tools such as **CloudWatch, SumoLogic, or DataDog**. - Build and maintain ingestion pipelines that land curated sensor and operational data into **Delta tables on Databricks** to support ML experimentation and model training. - Continuously optimize storage, partitioning, and query performance across large-scale datasets. - Participate in architecture reviews and ensure the platform adheres to security, compliance, and best-practice standards. **Skills and Qualifications** - **5+ years** of professional experience in software engineering or data engineering. - Strong programming skills in **Python** and **Golang**. - Deep understanding of **SQL** and modern data lake architectures (e.g., **Parquet, Iceberg, Delta Lake**). - Hands-on experience with **AWS** services including **S3, Lambda, Glue, Kinesis, Athena, and RDS**. - Proficiency with **Terraform** for automating infrastructure deployment and management. - Hands-on experience with **Databricks** for ML workflows: **MLflow** experiment tracking and model registry, model serving, and building training datasets in **Delta Lake (PySpark/Spark SQL)**. - Experience working with real-time or batch data ingestion at scale, and designing fault-tolerant **ETL/ELT** pipelines. - Familiarity with **DVC** for dataset versioning and management. - Familiarity with event-driven architectures and messaging systems like **Kafka** or **Kinesis**. - Strong debugging and optimization skills across cloud, network, and application layers. - Excellent collaboration, communication, and documentation skills. **Bonus Points** - Experience working with time-series or IoT sensor data at industrial scale. - Exposure to **gRPC** and **protobuf-based** data contracts. - Experience supporting ML pipelines and feature stores. - Working knowledge of **Kubernetes** concepts. - **Databricks M
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.