Senior Manager – Data Quality and Evaluation

Job not on LinkedIn

🕒 6 days ago

🇮🇪 Ireland – Remote

⏰ Full Time

🟠 Senior

📊 Data Scientist

👻 Ghost score 13%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Lion People Global

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2005

💼 Consulting

📣 Marketing

🎯 Recruiter

Consulting • Marketing • Recruitment

Lion People Global is a leading global growth solutions company that empowers organisations with integrated talent, M&A advisory, and consultancy services. It provides full-spectrum recruitment and executive search (fractional teams, direct hire, contingency, and leadership placement), strategic mergers & acquisitions support (seller incubation, deal navigation, investor matching, and exit planning), and tailored business consultancy and growth products — augmented by AI-enabled offerings and thought-leadership events to accelerate client expansion across international markets.

📋 Description

• Design and manage quality frameworks for AI data and evaluation programs • Translate customer requirements into quality standards, rubrics, acceptance criteria, review processes, and KPIs • Build scalable quality workflows for programs from pilot to production • Identify quality risks and resolve issues with delivery teams • Create repeatable processes for calibration, QA sampling, adjudication, reviewer performance tracking, and customer reporting • Support quality operations across multilingual evaluation, speech/audio QA, transcription, annotation, human preference evaluation, expert review, model response evaluation, coding evaluation, tool-use evaluation, and agent workflow evaluation • Create and improve rubrics, task instructions, reviewer guides, calibration exercises, golden datasets, and quality reporting templates • Lead calibration sessions with reviewers, annotators, quality specialists, delivery teams, and customer stakeholders • Define quality thresholds, error taxonomies, escalation rules, and corrective action plans • Monitor reviewer agreement, disagreement trends, error rates, contributor performance, and root causes of quality variance • Turn QA findings into improvements to instructions, training, tooling, staffing, and delivery workflows • Act as quality lead for strategic customer programs when needed • Support customer-facing quality readouts, pilot retrospectives, business reviews, escalations, and scale-up discussions • Provide data-backed reporting on quality performance, risks, corrective actions, and next steps • Partner with Program Management, Supply Chain, Solutions, Sales, and Operations to establish quality success • Define reviewer profiles, evaluator requirements, language requirements, domain expertise, onboarding needs, and performance expectations • Determine when programs require expert reviewers, QA leads, language leads, technical reviewers, or specialized evaluation talent • Build reusable quality assets, standardized approaches, scorecards, and sample evaluation frameworks • Improve visibility into quality performance across programs, reviewers, contributors, and workflows • Manage, coach, and support Quality Managers, Quality Leads, Quality Specialists, reviewers, and QA contributors • Identify hiring, training, and coverage needs • Create a culture of quality ownership, accountability, and continuous improvement • Anticipate and communicate team needs • Train and support team members, advocate for upskilling, and promote career growth • Provide support during increased workload periods and arrange coverage for sickness and absence

🎯 Requirements

• 5+ years of experience in quality operations, data operations, AI data services, localization quality, annotation quality, evaluation operations, trust and safety quality, or a related field • Experience managing quality programs for complex customer accounts or high-volume operational delivery • Strong understanding of QA methodologies, calibration, sampling, adjudication, error analysis, and performance reporting • Experience working cross-functionally with delivery, operations, supply chain, sales, and customer-facing teams • Strong analytical skills and ability to turn quality data into clear operational improvements • Excellent written and verbal communication skills, including ability to communicate quality issues clearly to customers and senior stakeholders • Comfort operating in fast-moving, ambiguous environments where processes are still being built • Strong people leadership skills with experience coaching quality specialists, reviewers, annotators, or operational teams • Nice to have: experience with AI data, RLHF, model evaluation, LLM evaluation, speech/audio evaluation, transcription, coding evaluation, multilingual evaluation, or expert review programs • Nice to have: experience designing rubrics, annotation guidelines, evaluation instructions, reviewer training, calibration workflows, or quality scorecards • Nice to have: experience supporting AI labs, enterprise AI teams, research teams, or technical customers • Nice to have: familiarity with human-in-the-loop data workflows, annotation platforms, QA tooling, dashboards, and data labeling operations • Nice to have: experience working with expert contributors, linguists, annotators, domain specialists, technical reviewers, or distributed talent networks • Nice to have: knowledge of multilingual evaluation, speech/audio QA, cultural appropriateness, or language-specific quality risks