Software Engineer, Model Runtime
OpenAI · San Francisco
About this role
**Software Engineer, Model Runtime** **About the Team** OpenAI's Hardware organization develops AI-native silicon and system-level solutions for advanced AI workloads, building on efforts like Jalapeño to create future generations of AI-native silicon and integrated systems. **About the Role** You will build the model runtime within the inference engine that executes frontier models at scale on OpenAI's custom silicon. The runtime translates demanding inference workloads into efficient execution while optimizing for throughput, latency, utilization, and reliability. **Key Responsibilities** • Design and implement LLM inference runtime for frontier models on custom silicon • Build scheduling, continuous batching, memory management, KV-cache management, and execution orchestration • Develop distributed execution strategies across chips, hosts, and racks • Optimize end-to-end latency, throughput, memory efficiency, and hardware utilization • Partner with kernel, compiler, architecture, and silicon teams to co-design interfaces • Enable new model features, execution patterns, and hardware capabilities • Create profiling, observability, benchmarking, and performance-modeling tools • Debug complex correctness, performance, and reliability issues across the stack **Ideal Candidate** • Strong systems programming experience (C++, Rust, Python, or similar) • Experience building runtimes, distributed systems, compilers, or model-serving infrastructure • Understanding of modern LLM inference, batching, KV-cache tradeoffs, and model parallelism • Ability to reason quantitatively about performance metrics and hardware utilization • Comfortable profiling and debugging across hardware-software stacks • Can design clean abstractions while maintaining low-level performance control • Experience working across model, systems, compiler, and hardware teams • Commitment to production quality and reliability at scale
Listing freshness
CronJobs last confirmed this listing 15h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.