AI Evaluation Specialist

🔥 15 hours ago

🌐 United States, Canada, +4 more countries – Remote

🇺🇸 United States – Remote 🇨🇦 Canada – Remote 🇬🇧 United Kingdom – Remote 🇮🇪 Ireland – Remote 🇦🇺 Australia – Remote 🇳🇿 New Zealand – Remote

🗽 New York – Remote

🗽 New York – Remote

💵 $30 - $90 / hour

⏱ Part Time

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

👻 Ghost score 0%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

24-MAG

WebsiteLinkedInAll Job Openings

2 - 10 employees

🤝 B2B

💼 Consulting

B2B • Consulting

24-MAG is a commercial strategy and execution firm that helps B2B organizations design and implement systems, workflows, and operating rhythms for sales, client management, and cross-functional projects. They focus on transforming scattered processes into aligned, measurable, and scalable commercial functions—covering pipeline structure, account management frameworks, and operational discipline for teams seeking efficient, intentional growth.

📋 Description

• Evaluate AI-generated outputs against detailed rubrics, guidelines, and defined quality standards • Assess responses for accuracy, relevance, completeness, reasoning quality, and adherence to instructions • Apply consistent and impartial judgement across large volumes of evaluation examples • Identify outputs that fail to satisfy important quality or task requirements • Maintain reliable assessment standards across repeated evaluation workflows • Identify reasoning gaps, logic errors, inconsistencies, unsupported conclusions, and tool-use failures • Analyse where AI-generated outputs diverge from expected reasoning or quality standards • Document recurring model weaknesses and opportunities for improvement • Produce clear, concise, and actionable written feedback on strengths and areas for improvement • Explain the reasoning behind evaluation decisions and quality scores • Maintain detailed, transparent, traceable, and reproducible assessment documentation • Participate in rubric interpretation and ambiguous-case discussions • Help refine assessment criteria as AI models and project requirements develop • Contribute insights supporting process optimisation and evaluation best practices • Collaborate with other reviewers to improve alignment and reliability across evaluation workflows

🎯 Requirements

• Experience in grading, quality assurance, editorial review, assessment, annotation, or another field requiring careful analysis and detailed feedback • Advanced, regular use of AI assistants such as ChatGPT, Claude, or comparable tools for professional work and productivity • Strong ability to synthesise complex information and communicate conclusions clearly in writing • Experience with process improvement, rubric development, operational quality assessment, or structured evaluation workflows is advantageous • Strong critical-thinking skills with particular emphasis on consistency, integrity, and fairness • High attention to detail and comfort reviewing large volumes of similar examples • Ability to work independently while maintaining consistent evaluation quality • Collaborative approach to discussing ambiguous cases and refining shared assessment standards • Excellent written English and professional documentation skills • Based in the United States, Canada, United Kingdom, Ireland, Australia, or New Zealand • Authorised to undertake contract work in the relevant country • No prior formal experience in AI research or model training is required

🏖️ Benefits

• Part-time independent contractor engagement • Fully remote work • Flexible project scope, workload, timing, and duration depending on project requirements