Senior Software Engineer - Model Evaluation & AI Systems
Deepgram · USA - Remote
About this role
## COMPANY OVERVIEW Deepgram is the leading platform underpinning the Voice AI economy, providing real-time APIs for speech-to-text (STT), text-to-speech (TTS), and production-grade voice agents at scale. Trusted by 1,300+ organizations and 200,000+ developers, Deepgram has processed 50,000+ years of audio and transcribed 1T+ words. ## THE OPPORTUNITY Deepgram is hiring a **Senior Software Engineer – Model Evaluation & AI Systems** to validate the quality of speech, audio, and multilingual models before they reach customers. This team builds the evaluation and quality assurance infrastructure—pipelines, harnesses, canaries, and test frameworks—to catch regressions, hallucinations, and quality issues early. You’ll define evaluation methodology, translate Research benchmarks into automated pass/fail gates, and build monitoring that keeps models honest in production. ## WHAT YOU’LL DO - Define and build evaluation methodologies for STT, TTS, and emerging LLM/RAG/agent/multimodal systems. - Design, build, and maintain automated evaluation pipelines for **batch and streaming** (e.g., **WER**, hallucination/runaway detection, latency, time-to-first-byte). - Build scalable, reproducible evaluation infrastructure (harnesses, orchestration, result aggregation) running against production models and large GPU clusters when needed. - Convert Research benchmarks and expected metrics into automated, enforceable pass/fail gates. - Build and operate canaries and continuous monitoring to detect quality regressions before customers are impacted. - Partner with DevOps/Infra to stand up ephemeral test environments and results-aggregation infrastructure. - Collaborate with Research, model training, inference, and product teams to provide trusted evaluation signals for release and optimization. - Integrate evaluation and quality gates into **CI/CD** for continuous verification. - Raise the bar through code reviews, technical design discussions, and strong engineering/QA practices. ## WHAT WE’RE LOOKING FOR - BS/MS/PhD in Computer Science, AI, Applied Math, or related field (or equivalent experience). - **5+ years** of professional software or QA engineering experience, with a track record shipping test infrastructure or evaluation systems. - Strong backend/scripting experience in **Python, Rust, Go,** or similar. - Experience designing/building automated test pipelines, evaluation frameworks, or data-processing systems. - Strong analytical skills—comfort reasoning about metrics, thresholds, and statistical variation (distinguish real regressions from noise). - Ability to lead through ambiguity and communicate effectively across research, engineering, and product teams. ## NICE TO HAVE / WAYS TO STAND OUT - Hands-on evaluation of modern AI systems (LLMs, RAG, agents, multimodal) including model behavior analysis. - Experience building tooling for broader access (e.g., **React Native** or other cross-platform frameworks). - Experience building/improving evaluation frameworks, benchmarks, or ML infrastructure used by other teams or external users. - Strong appreciation for evaluation quality: correctness, reproducibility, consistency. - Voice/audio/speech recognition or real-time systems experience; familiarity with **WER, MOS,** latency/TTFB. - Open-source involvement (contributions, reviews, maintenance, community engagement). - Experience acting as a technical bridge across teams/platforms. - Familiarity with cloud infrastructure, containerized/ephemera
Listing freshness
CronJobs last confirmed this listing 1h ago. If its source stops confirming the opening for seven days, this page is removed from active inventory.