AI Agent Evaluation Engineer

Job not on LinkedIn

🔥 12 minutes ago

🌐 Canada, United States – Remote

🇨🇦 Canada – Remote 🇺🇸 United States – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

👻 Ghost score 24%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Just posted.
• Jobs at this company usually fill in 62 days.
• No salary listed.

Apply Now

Find Similar Remote Jobs

Cresta

WebsiteLinkedInAll Job Openings

51 - 200 employees

☁️ SaaS

🤖 Artificial Intelligence

🏢 Enterprise

SaaS • Artificial Intelligence • Enterprise

Cresta is an enterprise-grade AI platform that focuses on enhancing contact center operations. By employing a unified platform for human and virtual agents, Cresta aims to improve customer experience, increase revenue, and reduce costs. The platform integrates AI to assist with sales, customer care, retention, and collections, providing real-time guidance and insights. Cresta's AI capabilities include conversation intelligence, agent assistance, quality management, and virtual agents. With a focus on automation and augmentation, Cresta seeks to transform workflows and customer interactions across various industries, including telecommunications, finance, and retail.

📋 Description

• Define and maintain evaluation rubrics covering adherence, outcomes, and conversation experience • Write annotation guidelines and failure taxonomies; run calibration sessions and measure inter-rater agreement • Classify and prioritize failure modes by severity and frequency • Review and annotate production conversations weekly, including multilingual evaluation • Trace failures to likely causes and route evidence to the appropriate owners • Build and operate evaluation pipelines combining automated checks, LLM-based judges, and human review • Calibrate automated judges against human labels • Design stratified sampling strategies • Build and maintain golden sets and regression suites • Maintain failure-mode coverage maps and close evaluation gaps • Design evaluations for high-stakes edge cases and upstream failures • Define quality and latency thresholds • Report quality trends and uncertainty to stakeholders and customers • Collaborate with forward-deployed engineers, backend engineers, product managers, and voice/language engineering teams • Document methodology for reproducible evaluation practices

🎯 Requirements

• 3+ years of professional experience in software engineering, applied ML, or data-intensive research roles • At least 1–2 years working hands-on with LLM-based systems in production • Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience • Built and operated an end-to-end evaluation system for a production LLM product • Deep understanding of evaluation methodology, including golden sets, judges, regression suites, and calibration • Understanding of LLM agents, including prompts, tool calls, context handling, and multi-turn state • Strong programming skills in Python and experience with LLM APIs • Ability to write rubrics and measure inter-rater agreement • Ability to design tests from open-ended quality questions • Willingness to listen to calls, label conversations, and build a failure taxonomy • Professional fluency in at least one language other than English • Strong written communication • Completion of a self-paced CoderPad technical assessment may be required after resume review

🏖️ Benefits

• Comprehensive medical, dental, and vision coverage with plans to fit you and your family • Flexible PTO to take the time you need, when you need it • Paid parental leave for all new parents welcoming a new child • Retirement savings plan to help you plan for the future • Remote work setup budget to help you create a productive home office • Monthly wellness and communication stipend to keep you connected and balanced • In-office meal program and commuter benefits provided for onsite employees • Equity as part of total compensation

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume