AI Quality & Evaluation Lead

🔥 4 minutes ago

🌽 Illinois – Remote

🌽 Illinois – Remote

💵 $150k - $200k / year

⏰ Full Time

🟠 Senior

🤖 AI Engineer

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Alight Solutions

WebsiteLinkedInAll Job Openings

10,000+ employees

🤝 B2B

🏥 Healthcare

☁️ SaaS

💰 $22.5M Post-IPO Secondary - Alight Solutions on 2023-08

B2B • Healthcare • SaaS

Alight Solutions is a leading benefits administration provider that delivers health, wealth, leave, and related HR point solutions for large organizations and their employees. The company helps manage benefits administration and human capital-related processes for enterprises, serving many of the world’s largest organizations and over 30 million people. Alight also emphasizes product and technology roles, corporate sustainability, total rewards, and a focus on transforming lives and organizations by maximizing health, financial and work wellbeing.

📋 Description

• Enables responsible and scalable AI adoption by defining technical quality standards, evaluation framework and control requirements across the AI lifecycle. • Partners closely with AI Engineering, Data Scientists, QA CoE, and Product teams to define quality and performance requirements. • Embeds quality considerations into model and system architecture from the onset. • Designs and maintains structured evaluation framework that assesses AI systems against defined quality bars. • Develops automated metrics for Generative AI performance. • Identifies and documents technical AI governance controls that enable automated compliance with performance and risk obligations.

🎯 Requirements

• 5–8+ years of experience in Data Science, ML Engineering, or AI Quality, with a focus on evaluation and statistical validation. • Practical experience partnering with engineers to design RAG, LLM-based Agents, or traditional ML pipelines. • Expert-level Python (Pandas, Scikit-learn) and experience with evaluation frameworks (e.g., RAGAS, TruLens, or MLflow). • Demonstrated ability to translate abstract trust concepts into mathematical metrics and enforceable technical controls. • Demonstrated ability to work without direct authority, driving quality adoption across engineering and product teams through enablement. • Ability to bridge the gap between high-level governance policy and low-level code implementation. • Bachelor’s degree in a technical field (e.g., Computer Science, Computer Systems Design) or equivalent professional experience.

🏖️ Benefits

• Health, dental and vision coverages starting Day One • Wellbeing programs • Retirement plans with contribution matching • Generous time off • Parental leave • Continuing education • Career growth opportunities