AI Evaluation Engineer

🔥 12 hours ago

🐊 Florida – Remote

🐊 Florida – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 10%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Sedgwick

WebsiteLinkedInAll Job Openings

10,000+ employees

🏗️ Construction

💼 Consulting

🏥 Healthcare

Construction • Consulting • Healthcare

Sedgwick is a global provider of technology-enabled risk, benefits, and integrated business solutions. They help people and organizations by managing and mitigating risk with solutions in accident, health, disability, unemployment compensation, and liability claims administration, among others. Sedgwick offers services such as claims administration, building consulting, forensic accounting, and forensic engineering. Their specialties include property restoration, brand protection, and loss prevention across several industries, including agriculture, construction, and environmental sectors. The company emphasizes diversity, equity, and inclusion (DEI) as well as environmental, social, and governance (ESG) practices.

📋 Description

• Design, build, and operate evaluation systems for GenAI solutions • Establish standardized, automated evaluation frameworks for LLMs and traditional ML models • Deploy observability solutions for model drift, hallucination risk, latency, throughput, and token consumption • Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications • Build reusable evaluation frameworks, templates, and libraries • Translate business requirements into measurable quality criteria and acceptance thresholds • Develop automated evaluation pipelines integrated into CI/CD with regression testing and quality gates • Implement offline and online evaluation, including benchmarks, golden datasets, production monitoring, sampling, drift, and quality alerts • Support A/B and shadow testing for model, prompt, and architecture changes • Design and maintain LLM-as-a-judge evaluators with rubric design, bias mitigation, and human-label calibration • Establish human annotation workflows and labeling guidelines • Build golden datasets, adversarial and edge-case suites, and synthetic test-data generation approaches • Apply data-handling practices including de-identification, synthetic data, and access controls • Implement red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing • Produce evaluation evidence and reporting for security, privacy, compliance, and model-risk reviews • Contribute to responsible AI standards and release-readiness criteria • Embed with product pods to onboard teams to evaluation practices and coach them toward ownership • Build dashboards and reporting for quality and cost/latency trends • Evaluate and recommend evaluation tooling and platforms; document standards and best practices • Perform other duties as assigned • Travel as required

🎯 Requirements

• Master’s degree in a computer science or a quantitative discipline from an accredited college or university preferred • 4+ years in software, ML, or data engineering • At least 1-2 years working hands-on with LLM-based applications • Professional experience with evaluation and observability frameworks like LangSmith, Arize Phoenix, or WhyLabs • Strong understanding of statistical validation and model evaluation metrics • Strong Python skills and software engineering fundamentals, including testing, version control, CI/CD, and code review • Practical experience evaluating LLM/RAG systems, including retrieval metrics such as recall, precision, MRR/nDCG and generation metrics • Hands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation • Working knowledge of statistics and experimental design, including sampling, confidence intervals, significance testing, and inter-rater agreement • Familiarity with at least one eval or observability tool, such as Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or equivalent • Experience with cloud platforms such as Azure, AWS, or GCP and their managed LLM services • Clear communication skills and ability to work with engineers, product owners, SMEs, and risk stakeholders • Excellent oral and written communication skills, including presentation skills • Ability to work in a team environment • Ability to meet or exceed Performance Competencies • Preferred: experience evaluating agentic systems, background in ML evaluation/search/relevance/NLP/QA automation, LLMOps/monitoring/observability experience, familiarity with AI risk and governance frameworks, data annotation workflows, and enterprise controls

🏖️ Benefits

• Work-life balance • Reasonable accommodations when applicable and appropriate • Equal Opportunity Employer • Drug-Free Workplace

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume