AI Quality Analyst, Personalization, English (US)

Job not on LinkedIn

🔥 19 minutes ago

🇺🇸 United States – Remote

⏳ Contract/Temporary

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

Where this job comes from

Posted on Careerflow.ai's careers page ↗Careerflow.ai's careers page, not through a job board

First seen by Remote Rocketship 19 minutes ago

Apply Now
Logo of Careerflow.ai

Careerflow.ai

11 - 50 employees

Founded 2022

🤖 Artificial Intelligence

👥 HR Tech

☁️ SaaS

💰 $700k PRE-SEED on 2024-05

Artificial Intelligence • HR Tech • SaaS

Careerflow. ai is an AI-powered job search platform that helps users create and optimize resumes and cover letters, track applications, improve LinkedIn profiles, and prepare for interviews. It also provides career tools to organizations such as university career centers, bootcamps, and outplacement providers.

📋 Description

• Evaluate a new Gemini personalization feature using information from past Gemini conversations, Gmail, Google Search, and YouTube activity • Design and execute multi-turn conversational prompts, typically 1–5 turns, using personal information and experiences • Evaluate whether model responses appropriately apply personalization based on the prompt intent • Analyze responses for Grounding issues, unsupported claims, flawed inferences, and hallucinations • Assess Integration quality and whether personal data is incorporated naturally without robotic overnarrating • Stack-rank two model responses side-by-side to determine which is more helpful, easy to use, and enjoyable • Write clear, defensible rationales referencing specific conversation turns • Extract and verify Debug Info to confirm proper use of chat summaries and data sources • Maintain data hygiene by deleting evaluation conversations

🎯 Requirements

• Ability to read and write English with a high degree of competence • Willingness to use a primary personal Google account and enable personal data sources • Full-time availability in the local time zone • Ability to evaluate nuanced and ambiguous AI responses • Experience designing creative, multi-turn prompts based on personal context • Understanding of personalization concepts, including incorrect personalization, poor inferences, and forced connections • Ability to review Side-by-Side (SxS) model responses and identify subtle differences in naturalness and overnarrating • Excellent written communication and ability to write clear, concise, structured rationales referencing turn numbers • Ability to provide constructive feedback and detailed annotations • Excellent communication and collaboration skills • Ability to work independently in a remote setting • Desktop or laptop with a good internet connection • BS/BA degree or equivalent experience in a relevant field • Experience in data annotation, AI quality evaluation, content moderation, or a related role strongly preferred • Minimum commitment of 30 hours per week, with at least 4 hours per day and 4 hours of overlap with PST

🏖️ Benefits

• Contractor engagement • Full-time availability with 30- or 40-hour weekly commitment options • Remote work arrangement • Flexible scheduling within the candidate's local time zone, with 4 hours of overlap with PST

Apply Now

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

🔥 2 hours ago

Weekday (YC W21)

11 - 50

💼 Consulting

👥 HR Tech

☁️ SaaS

P&C insurance professionals converting underwriting, reinsurance, pricing, reserving, and claims expertise into AI training datasets. Fully remote, flexible contractor project.

🔥 2 hours ago

Weekday (YC W21)

11 - 50

💼 Consulting

👥 HR Tech

☁️ SaaS

Practicing economists creating and evaluating training data for advanced AI systems. Focusing on forecasting, antitrust, trade policy, public policy, and economic impact analysis.

🕒 2 days ago

Job Mobz

51 - 200

🎯 Recruiter

👥 HR Tech

🤝 B2B

AI automation specialist building workflows and agents for Employer.com, a provider of back-office business solutions. Improving content operations, productivity, and digital distribution.

🕒 2 days ago

Mindrift

501 - 1000

🤖 Artificial Intelligence

🏪 Marketplace

☁️ SaaS

US-licensed physicians evaluating EHR clinical cases for Mindrift’s AI testing projects. Scoring diagnostic reasoning, answers, and distractors through flexible task-based contractor work.

🕒 2 days ago

Mindrift

501 - 1000

🤖 Artificial Intelligence

🏪 Marketplace

☁️ SaaS

US-licensed physicians reviewing clinical EHR vignettes and reasoning quality. Mindrift uses contributor feedback to evaluate and improve medical AI systems.

Developed by Lior Neu-ner. I'd love to hear your feedback — Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok