JSON & Rubric Evaluation Specialist

Job not on LinkedIn

🔥 7 minutes ago

🗽 New York – Remote

🗽 New York – Remote

💵 $45 - $170 / hour

⏱ Part Time

🟡 Mid-level

🟠 Senior

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

24-MAG

WebsiteLinkedInAll Job Openings

2 - 10 employees

🤝 B2B

💼 Consulting

B2B • Consulting

24-MAG is a commercial strategy and execution firm that helps B2B organizations design and implement systems, workflows, and operating rhythms for sales, client management, and cross-functional projects. They focus on transforming scattered processes into aligned, measurable, and scalable commercial functions—covering pipeline structure, account management frameworks, and operational discipline for teams seeking efficient, intentional growth.

📋 Description

• Review written instructions, interpret JSON-based responses, and assess each output independently using predefined rubrics • Read, parse, and interpret AI-generated outputs presented in JSON format • Understand nested fields, structured values, and relationships between output components • Identify missing, inconsistent, malformed, or incorrectly represented information • Assess whether each response satisfies the underlying task instructions • Review detailed task descriptions and corresponding grading criteria • Evaluate each JSON output against predefined rubric requirements • Apply scoring standards consistently across high-volume task batches • Distinguish minor issues from material failures based on the provided criteria • Document grading decisions clearly where explanations are required • Maintain strong attention to detail across repetitive evaluation workflows • Identify edge cases, ambiguous outputs, and subtle inconsistencies • Follow project instructions precisely without introducing unsupported assumptions • Escalate unclear rubric language or conflicting task requirements appropriately • Incorporate calibration feedback into subsequent evaluations • Complete all assigned assessments without assistance from AI tools • Manage tasks independently within short turnaround windows • Review completed work before submission for accuracy and completeness • Maintain reliable quality throughout the duration of the sprint

🎯 Requirements

• Experience reading and interpreting JSON files or JSON-formatted data • Familiarity with structured grading rubrics and evaluation criteria • Prior experience as a grader, reviewer, rater, annotator, or quality evaluator • Strong written reasoning and attention to detail • Ability to apply complex instructions consistently across multiple tasks • Comfort working within fast-paced, high-volume evaluation workflows • Availability to contribute approximately 20–40 hours per week • Ability to complete assigned tasks within approximately 12–24 hours • Current residence in the United States

🏖️ Benefits

• Weekly payments via Stripe or Wise