AI Quality Assurance – Evaluation Specialist

🔥 1 minute ago

🌐 Brazil, Argentina, +4 more countries – Remote

🇧🇷 Brazil – Remote 🇦🇷 Argentina – Remote 🇨🇴 Colombia – Remote 🇨🇱 Chile – Remote 🇵🇪 Peru – Remote 🇺🇾 Uruguay – Remote

⏳ Contract/Temporary

🟡 Mid-level

🟠 Senior

🔧 QA Engineer (Quality Assurance)

👻 Ghost score 12%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Gramian Consulting

WebsiteLinkedInAll Job Openings

2 - 10 employees

Founded 2025

💼 Consulting

📦 Logistics

📣 Marketing

Consulting • Logistics • Marketing

Gramian Consulting is a remote-first consulting firm that connects engineering and data/AI talent with organizations through talent augmentation, recruiting, dedicated teams, and contractor management. The firm provides Data & AI services including LLM training and fine-tuning, AI agents and assistants, MLOps, and AI infrastructure, and it offers mentorship and education programs for career readiness, interview preparation, and international market orientation. Rooted in hands-on engineering and recruiting experience, Gramian helps clients scale technical teams and extract business value from AI while developing individual talent.

📋 Description

• Validate task instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness. • Review AI agent execution traces, tool calls, and generated deliverables to assess whether outcomes are justified. • Audit grading logic to identify brittle checks, incorrect expected answers, and unsupported rubric criteria. • Identify cases where valid alternative solutions are unfairly penalized. • Investigate discrepancies between model performance, task defects, grader errors, and environment or tool failures. • Independently assess automated QC findings rather than accepting them without verification. • Document concise, evidence-backed findings and actionable recommendations. • Clearly communicate uncertainty and distinguish confirmed issues from potential problems. • Verify that task revisions resolve previously identified defects. • Collaborate with relevant teams to improve evaluation quality and reliability.

🎯 Requirements

• Experience in technical QA, AI evaluation, data analysis, software testing, or a related technical field. • Comfortable reading and interpreting Python, SQL, shell scripts, structured data, and execution logs. • Strong analytical skills, including the ability to validate calculations and reconcile conflicting evidence. • Experience assessing the correctness and completeness of technical or professional deliverables. • Strong written English and the ability to provide specific, reproducible feedback. • High attention to detail when reviewing task requirements, grading behavior, and model outputs. • Ability to independently investigate issues and make evidence-based decisions.