Forward Deployed Engineer, Inference
Cohere · United States
About this role
## Cohere — Staff Forward Deployed Engineer (Inference) ### About the role Cohere is a security-first enterprise AI company building frontier foundation models and end-to-end products for real-world business problems. In this role, you’ll help bridge the gap between frontier AI capability and complex customer environments. As a **Staff Forward Deployed Engineer (FDE)**, you’ll act as a principal technical translation layer between Cohere engineering teams and strategic enterprise partners. You’ll own the architecture for highly ambiguous, customer-specific inference challenges—ranging from **multi-tenant model serving** to **multi-datacenter disaster recovery** and **custom hardware validation**—and lead technical adoption end-to-end to achieve **low latency**, **high throughput**, and **seamless resilience** in the field. ### What you’ll do - Embed with customer engineering teams to design the “connective tissue” between Cohere models and production environments. - Design advanced model serving topologies, including: - multi-tenant architectures - intelligent model routing - cross-datacenter failover for multi-cluster setups - Optimize “AI-in-a-box” and private deployments, balancing localized SRE constraints (e.g., Kubernetes multi-cluster scale) with model performance. - Benchmark and profile customer LLM workloads; root-cause performance degradation using metrics, dashboards, and distributed logs. - Accelerate hardware validation by collaborating with clients to run and support Cohere models on next-gen accelerators (GPUs, TPUs, AMD, and custom ASICs). - Translate field friction into platform upgrades—providing prescriptive feedback and contributing to core product features. - Ensure Day-1 production reliability by helping customers establish deployment patterns, SLOs, and observability frameworks. ### You may be a good fit if you have - **8+ years** of experience running production infrastructure, designing large-scale distributed systems, or deploying enterprise software in complex environments. - Deep technical depth in **production Kubernetes**, including orchestration across multi-cluster and hybrid/multi-cloud ecosystems (GCP, Azure, AWS, OCI). - Strong understanding of **LLM inference dynamics**, including how prefill (compute-bound) and decode (memory-bound) phases impact live performance. - Hands-on experience with modern inference engines/optimization stacks (e.g., **vLLM**, **TensorRT-LLM**, **lmd**, or **Dynamo**). - Familiarity with distributed serving techniques such as: - disaggregated prefill/decode serving - distributed KV caching - multi-GPU workloads - Exceptional customer-facing technical leadership, with a track record of scoping ambiguous problems and steering executive technical stakeholders. - High agency and the ability to write same-day production code while navigating client-side deployment politics/blockers. - Strong programming foundations in **Python, Go, or C++**; comfortable moving between backend data pipelines and infrastructure tuning. ### Perks (full-time employees) - Weekly lunch stipend: **$75/£75** (or equivalent) - Full health and dental benefits, including a mental health budget - RRSP matching / 401K / Pension Scheme - **100% Parental Leave top-up** for up to 6 months (either parent) - Annual enrichment benefits (arts & culture, fitness/wellness, quality time, workspace credit) - Education & learning stipend (conferences, courses, coaching) - **6 weeks paid vacation** (30 wo
Listing freshness
CronJobs last confirmed this listing 58m ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.