Data Scientist – Agent Evaluations, Quality

Job not on LinkedIn

🔥 0 minutes ago

🏄 California – Remote

🏄 California – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

📊 Data Scientist

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Clera

WebsiteLinkedInAll Job Openings

1 - 10 employees

Founded 2025

💰 $180k Grant on 2025-04

Clera is a company that displays an onboarding/setup message to users: "Please wait while we prepare everything for you. " Based on this single line of text, the company's specific product, service, or industry cannot be determined with confidence. The message suggests Clera provides a user-facing digital service or application that performs setup or provisioning, but there is insufficient information to identify a precise industry.

📋 Description

• Architect and maintain automated evaluation pipelines measuring agent quality across product surfaces • Translate product capabilities into pass, partial-pass, and failure criteria for complex multi-step tasks • Build gold datasets and regression suites covering workflows, edge cases, and adversarial scenarios • Define and track task success, tool-selection accuracy, instruction adherence, factual consistency, latency, cost, and reliability metrics • Design deterministic and model-based graders, calibrate LLM-as-a-judge systems, and monitor grader agreement • Compare models, prompts, and implementations using offline experiments and production evidence • Analyze traces and production outcomes to identify root causes and build a failure taxonomy • Convert production failures into regression cases and close evaluation-coverage gaps • Build dashboards and release-quality signals for engineering, product, and leadership • Partner with capability engineers to recommend improvements and verify fixes without unacceptable regressions

🎯 Requirements

• 4+ years in Applied Data Science or Machine Learning roles focused on evaluation systems, automated data pipelines, or production ML infrastructure • Experience designing and implementing automated evaluation frameworks, success criteria, and regression suites for complex AI/ML or agentic systems • Production-grade proficiency in Python and SQL • Experience building and maintaining automated analytical pipelines on large datasets • Experience applying significance testing, variance analysis, and sampling to non-deterministic AI/ML systems • Experience developing labeled datasets, annotation guidelines, and quality-control processes for ground-truth data • Deep understanding of LLM agent behaviors, including tool use, multi-step execution, retrieval, and failure modes • Ability to analyze model traces, tool calls, and outputs across model, prompt, tool, and data layers • Experience using production telemetry and observability data to monitor quality, build dashboards, and analyze user outcomes • Nice to have: LLM-as-a-judge, model-based grading, or AI benchmarking experience • Nice to have: production ML products, agentic systems, or customer-facing consumer software experience • Nice to have: public research benchmarks or academic evaluation methodology experience • Visa sponsorship is not available for this role