AI Quality Analyst

Job not on LinkedIn

🕒 April 9

Apply Now

Find Similar Remote Jobs

Receive Emails For Remote Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Shyft6

WebsiteLinkedInAll Job Openings

201 - 500 employees

Founded 2019

👥 HR Tech

🎯 Recruiter

🤝 B2B

HR Tech • Recruitment • B2B

Shyft6 is a trusted partner for innovative companies, specializing in workforce solutions that encompass staffing and recruiting, workforce management, and human capital strategy. With over 26 years of combined experience, Shyft6 offers a comprehensive range of services including contingent workforce supplier management, recruitment process outsourcing, staff augmentation, executive search, and multi-state payroll. Their focus is on empowering organizations through effective workforce development and project management, ensuring successful delivery and skilled talent acquisition.

📋 Description

• Design and execute structured evaluations for AI-enabled features and workflows. • Assess outputs for groundedness, instruction adherence, consistency, usefulness, tone, control compliance, and risk. • Identify hallucinations, unsupported assertions, missing logic, and unsafe recommendations. • Build and maintain golden datasets, benchmark prompts, comparison sets, and scorecards. • Develop rubrics that allow quality to be measured consistently across releases and changes. • Compare performance across prompt versions, workflow revisions, tools, and models. • Support release decisions with evidence on quality regression or improvement. • Work closely with Finance SMEs, product managers, and engineers to determine what acceptable looks like in real business contexts. • Help define human-review thresholds and escalation patterns for higher-risk use cases. • Analyze reviewer feedback, override patterns, and live quality signals to improve evaluation coverage over time.

🎯 Requirements

• 4+ years of experience in QA, analytics, business process validation, AI evaluation, operations, or similar roles. • Strong writing, analysis, and pattern-recognition skills. • Experience evaluating outputs against nuanced criteria rather than only binary correctness. • Ability to work with structured rubrics, scenario libraries, and evidence-based reviews. • Comfort collaborating across Engineering and business teams. • Experience with finance, accounting, FP&A, transaction services, or business process design preferred. • Bachelor's degree preferred.

🏖️ Benefits

• Salary plus performance-based bonus