Lead AI Trainer – Prompt Evaluator

3 days ago

Apply Now

Receive Emails with Similar Jobs

Social Discovery Group

WebsiteLinkedInAll Job Openings

Social Impact • Media • Entertainment

Social Discovery Group is the world's largest group of social discovery companies, focused on solving the problems of loneliness, isolation, and disconnection through digital reality. With 500 million users across 150 countries, the group boasts over 60 global communication and entertainment platforms. Employing a team of 1200 professionals, the company organizes events that unite a global community of professionals and provides career opportunities for individuals seeking ambitious challenges.

1001 - 5000 employees

Founded 20 years

🌍 Social Impact

📱 Media

📋 Description

• Review and assess the quality of datasets produced by the content team, ensuring accuracy, consistency, and adherence to labeling guidelines. • Provide constructive feedback and help maintain a high quality bar across annotators. • Personally perform high-quality annotation and dataset creation as a subject-matter expert. • Validate conversational AI model outputs (e.g., SbS evaluations, error tagging, scenario stress-testing) to detect issues before production release. • Work closely with the ML team to refine instructions, dataset specifications, and evaluation criteria for model training cycles. • Identify recurring issues in annotations or model behavior and suggest improvements to workflows, QA processes, or validation pipelines. • Contribute insights on prompts and data strategies to accelerate model quality improvements.

🎯 Requirements

• Experience in dataset annotation and content validation for conversational AI / chat models / NLP products. • Hands-on experience with SbS (side-by-side) or other model evaluation frameworks (e.g., quality scoring, tagging, error categorization). • Ability to analyze conversational responses and identify accuracy, tone, safety, and contextual relevance issues. • Experience preparing or following annotation guidelines and QA checklists. • Strong attention to detail and ability to consistently apply evaluation criteria. • Ability to provide clear and constructive written feedback to other annotators. • Familiarity with prompt engineering practices is a plus. • Familiarity with LLM-based automated evaluators / AI-as-a-judge is a plus. • English proficiency (B2+) in reading and writing.

🏖️ Benefits

• REMOTE OPPORTUNITY to work full time; • Vacation 28 calendar days per year; • 7 wellness days per year (time off) that can be used to deal with household issues, to lie down and recover without taking sick leave; • Bonuses up to $5000 for recommending successful applicants for positions in the company; • 50% payment for professional training, international conferences and meetings ; • Corporate discount for English lessons; • Health benefits. According to the paychecks, if you are not eligible for corporate medical insurance, the company will compensate you with up to $ 1,000 gross per year per employee. This can be spent on self-purchase of health insurance or on doctor’s fees for yourself and close relatives (spouse, children); • Workplace organization. The company provides all employees with an equipped workplace and all the necessary equipment (table, armchair, wifi, etc.) in our offices or co-working locations. In the other locations, the company provides reimbursement of workplace costs up to $ 1000 gross once every 3 years, according to the paychecks. This money can be spent on the rent of the co-working room, on equipping the working place at home (desk, chair, Internet, etc.) during those 3 years; • Internal gamified gratitude system: receive bonuses from colleagues and exchange them for our merchandise, team building activities, massage certificates, etc.