AI Evaluation Engineer

Job not on LinkedIn

🔥 0 minutes ago

⛰️ Colorado – Remote

⛰️ Colorado – Remote

💵 $120k - $170k / year

⏰ Full Time

🟡 Mid-level

🟠 Senior

🤖 AI Engineer

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

GovWorx

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2023

🤖 Artificial Intelligence

🏛️ Government

☁️ SaaS

🔥 Funding within the last year

💰 Private equity on 2025-07

Artificial Intelligence • Government • SaaS

GovWorx is a software company that delivers AI-powered solutions to support emergency communications centers and other first responder agencies. Their platform offers real-time call guidance and translation, automated quality assurance, pre-hire assessments, AI-driven simulations and training, live transcription, and analytics to improve performance, consistency, and retention for 9-1-1 telecommunicators, law enforcement, and EMS professionals. GovWorx integrates with CAD, radio, and phone systems to streamline QA/QI, coaching, and operational workflows for public safety organizations.

📋 Description

• Design, build, and maintain automated AI evaluation pipelines for production LLM applications • Develop prompt engineering strategies and iterate on prompts and compare LLMs using quantitative evaluation methods • Build offline evaluation datasets and regression testing frameworks to measure AI performance over time • Analyze production AI behavior using Python, SQL, and statistical techniques to identify opportunities for improvement • Design experiments, A/B tests, and benchmarking methodologies for evaluating prompt and model changes • Develop dashboards and reporting that communicate AI quality, reliability, and performance metrics • Partner with engineering and product teams to safely deploy and monitor improvements to production AI systems • Investigate model failures through detailed error analysis and recommend improvements to prompts, evaluation datasets, and workflows • Help establish best practices for Responsible AI, evaluation methodologies, and continuous model improvement

🎯 Requirements

• Must have US citizenship and pass FBI fingerprint and background check in multiple states • 3+ years of experience in software engineering, machine learning, data science, or a related technical field • Experience designing evaluation metrics and interpreting AI model performance • Understanding of statistical methods including hypothesis testing and experiment design • Strong Python development experience • Strong SQL skills with experience analyzing large datasets • Experience building or supporting production LLM or Generative AI applications • Experience with prompt engineering and systematic prompt evaluation

🏖️ Benefits

• Offers Bonus • Benefits (Health, Dental, Vision) • Flexible Time Off