Senior Software Engineer – Model Evaluation, AI Systems

🔥 1 hour ago

🇺🇸 United States – Remote

💵 $180k - $240k / year

⏰ Full Time

🟠 Senior

⚙️ Systems Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Deepgram

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2015

💼 Consulting

🏥 Healthcare

📦 Logistics

💰 $47M Series B on 2022-11

Consulting • Healthcare • Logistics

Deepgram is a leading voice AI company that provides powerful APIs for speech-to-text, text-to-speech, and language understanding applications. Their platform enables developers to build sophisticated voice AI solutions for use cases such as contact centers, medical transcription, conversational AI, and more. Known for unmatched accuracy, speed, and cost-effectiveness, Deepgram's technology is trusted by top enterprises and startups worldwide. By offering real-time and highly accurate transcription capabilities, Deepgram helps businesses gain insights from voice data, making it an essential tool for transforming voice interactions.

📋 Description

• Define and build evaluation methodologies for Deepgram's models, spanning speech-to-text, text-to-speech, and emerging LLM, RAG, agent, and multimodal systems. • Design, build, and maintain automated evaluation pipelines across batch and streaming (e.g. WER, runaway/hallucination detection, latency and time-to-first-byte), with a focus on correctness, reproducibility, and ease of adoption. • Build scalable, reproducible evaluation infrastructure — harnesses, orchestration, and result-aggregation pipelines — running against production models and, where needed, large GPU clusters. • Translate Research benchmarks and expected model metrics into automated, enforceable pass/fail gates. • Build and operate canaries and continuous-monitoring systems that detect quality regressions in production before they reach customers. • Partner with DevOps/Infra to stand up ephemeral test environments and results-aggregation infrastructure. • Work alongside Research, model training, inference, and product teams to provide trusted evaluation signals that inform release and optimization decisions. • Integrate evaluation and quality gates into CI/CD so quality is verified continuously, not manually. • Help raise the bar through code reviews, technical design discussions, and strong engineering and QA practices.

🎯 Requirements

• BS, MS, or PhD in Computer Science, AI, Applied Math, or a related field, or equivalent experience. • 5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems (senior candidates with significantly deeper experience welcome). • Solid backend/scripting experience in a language such as Python, Rust, Go, or similar. • Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems. • Strong analytical skills and comfort reasoning about metrics, thresholds, and statistical variation in results — able to distinguish real regressions from noise. • Ability to take charge of ambiguous technical challenges and communicate effectively across research, engineering, and product teams.

🏖️ Benefits

• Offers Equity • Offers Bonus • 10% Annual Bonus