AI Quality & Evaluation Lead

Job not on LinkedIn

🔥 2 hours ago

🌽 Illinois – Remote

🌽 Illinois – Remote

💵 $150k - $200k / year

⏰ Full Time

🟠 Senior

🤖 AI Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Alight Solutions

WebsiteLinkedInAll Job Openings

10,000+ employees

Founded 2016

💼 Consulting

🏥 Healthcare

📦 Logistics

Consulting • Healthcare • Logistics

Alight Solutions is a leading cloud-based human capital technology and services provider serving some of the world's largest organizations and over 35 million people and dependents. The company focuses on transforming lives and organizations by enhancing health, financial, and work wellbeing. Alight offers a wide range of services, including customer care, technology roles, consulting, application services, and business support, with opportunities for virtual or work-from-home positions. The company emphasizes corporate social responsibility and providing a positive impact on clients, colleagues, and communities.

📋 Description

• Partnering directly with AI Engineers, Application Developers and Data Scientists during the design phase to define technical quality acceptance criteria and fit-for-use requirements • Embedding quality considerations into model and system architecture from the onset, specifically for complex patterns like RAG and autonomous Agents • Defining golden truth requirements and evaluation dataset standards; partner with Data Science teams to ensure datasets reflect production-level complexity • Defining quality and evaluation expectations for third-party AI systems and vendor-supplied models, ensuring consistent governance standards regardless of model origin • Designing and maintaining structured evaluation framework that assesses AI system against defined quality bars (e.g. Goodness-of-Fit, Calibration, Stability) • Developing automated metrics for Generative AI performance including Groundedness (Hallucination detection), Faithfulness, Completeness, and other domain-relevant metrics • Defining and operationalize fairness and bias evaluation criteria, including demographic parity assessments and disparate impact testing for client-facing AI systems • Calibrating evaluation thresholds and monitoring cadence to AI risk tier, ensuring proportionate controls without over-engineering lower-risk use cases • Identifying and document technical AI governance controls that enable automated compliance with performance and risk obligations • Establishing drift and ongoing monitoring requirements, defining statistical triggers for feature and concept drift that necessitate model intervention • Providing objective, data-driven evaluation outputs that support AI governance reviews and risk classification • Maintaining authoritative documentation of AI controls to support audit, regulatory review, and internal assurance activities

🎯 Requirements

• 5–8+ years of experience in Data Science, ML Engineering, or AI Quality • Practical experience partnering with engineers to design RAG, LLM-based Agents, or traditional ML pipelines • Expert-level Python (Pandas, Scikit-learn) • Experience with evaluation frameworks (e.g., RAGAS, TruLens, or MLflow) • Demonstrated ability to translate abstract trust concepts into mathematical metrics and enforceable technical controls • Ability to bridge the gap between high-level governance policy and low-level code implementation • Bachelor’s degree in a technical field (e.g., Computer Science, Computer Systems Design) or equivalent professional experience

🏖️ Benefits

• health, dental and vision coverages starting Day One • wellbeing programs • retirement plans with contribution matching • generous time off • parental leave • continuing education • career growth opportunities • flexible working arrangements